fix(ci): rebuild broken dev module tags (GitHub + GitLab)#2683
Merged
Conversation
The recheck-and-rebuild workflow deleted only the images-digests and prepare-bundle stages before rebuilding. werf stage tags are content-based and stable, so deleting a stage forces only that stage to rebuild without invalidating any downstream signature. bundle kept its cached signature, so the build action republished :tag from the stale bundle stage and its images_digests.json still pointed at garbage-collected component manifests (run 29413768115 failed the re-check with 4 missing digests). Delete the full baking chain images-digests -> prepare-bundle -> bundle so the fresh images_digests.json propagates into the published module :tag. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The recheck-and-rebuild workflow deleted only the images-digests chain by digest. But a garbage-collected component keeps a resolvable stage tag while its final manifest is gone, so werf reused the stage and re-reported the dead digest, and delete-by-digest failed with MANIFEST_UNKNOWN. The regenerated images_digests.json kept pointing at the same collected digests and the recheck still failed (runs 29413768115, 29416126578). For every component whose final image is no longer pullable, delete its stages by tag so werf can no longer find the stage by signature and rebuilds and re-pushes a live image; then regenerate images-digests and prepare-bundle so the fresh digests reach the module :tag. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Check and Re-check ran the same registry-pullability scan inline. Extract it to .github/scripts/bash/check-and-rebuild/check-image-digests.sh and call it from both steps. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Give the check script --repo/--tag flags with named vars, and move the broken-stage deletion into rebuild-broken-stages.sh (--repo/--report) so the workflow steps are thin calls. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Per-image OK/miss lines, a summary with the missing images and totals. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The job checks out the released tag, whose tree predates these scripts, so calling them from the working directory failed with 'No such file' (exit 127). Check out the workflow ref first, copy the scripts to $RUNNER_TEMP, then check out the tag and call them from there. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Port dev_check-and-rebuild to GitLab: a manual (web) job, triggered with a TAG variable, that rechecks a published dev tag and rebuilds it by deleting the broken components' stages by tag. Helper scripts live under .gitlab/ci/scripts/bash/check-and-rebuild/, mirroring the GitHub layout. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
universal-itengineer
changed the title
universal-itengineer
marked this pull request as ready for review
July 15, 2026 14:15
universal-itengineer
requested review from
Isteb4k and
nevermarine
as code owners
July 15, 2026 14:15
nevermarine
approved these changes
Jul 15, 2026
deckhouse-BOaTswain
pushed a commit
that referenced
this pull request
Jul 16, 2026
Description The manual "check and rebuild" workflow is meant to repair a published dev module tag after dev-registry GC collects some of its component images. It didn't actually repair anything: werf kept reusing the collected stages (their tags still resolve) and re-reported the same dead digests, so the rebuilt tag was still broken. Now, for every component whose final image is no longer pullable, the workflow deletes its stages by tag (the dead digest can't be deleted), forcing werf to rebuild and re-push a live image; images-digests and prepare-bundle are dropped too so images_digests.json is regenerated and the tag republished. The check/rebuild logic moved into reusable scripts, and the same flow is ported to GitLab CI. --------- Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The manual "check and rebuild" workflow is meant to repair a published dev module tag after dev-registry GC collects some of its component images. It didn't actually repair anything: werf kept reusing the collected stages (their tags still resolve) and re-reported the same dead digests, so the rebuilt tag was still broken.
Now, for every component whose final image is no longer pullable, the workflow deletes its stages by tag (the dead digest can't be deleted), forcing werf to rebuild and re-push a live image;
images-digestsandprepare-bundleare dropped too soimages_digests.jsonis regenerated and the tag republished. The check/rebuild logic moved into reusable scripts, and the same flow is ported to GitLab CI.Why do we need it, and what problem does it solve?
A dev tag whose component images were garbage-collected can't be deployed. This is the tool to fix such a tag — and until now it reported success while leaving it broken.
What is the expected result?
Run against a tag with GC'd component images: after the rebuild the recheck reports all component images pullable and the job passes. Verified on the DEV registry (run 29418170976 repaired the tag; a follow-up run then found it healthy).
Checklist
Changelog entries