fix(ci): stop a cancelled deploy from taking a GHCR lease it cannot release - #3032
Queued
devantler wants to merge 1 commit into
Queued
fix(ci): stop a cancelled deploy from taking a GHCR lease it cannot release#3032devantler wants to merge 1 commit into
devantler wants to merge 1 commit into
Conversation
Contributor
✅
|
Contributor
Author
@coderabbitai review |
Contributor
|
|
Contributor
|
Warning Review limit reached
Next review available in: 40 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
CodeRabbit reported @codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Contributor
Author
ReadinessProgrammatically tested. RED first: the new contract test failed naming the exact offending step, and the existing assertion failed on Reviewed. Codex green at this exact head ( Exercised. Two things I could observe directly, and one I could not:
Note on merging. |
devantler
marked this pull request as ready for review
August 9, 2026 15:37
This was referenced Aug 9, 2026
devantler
enabled auto-merge
August 9, 2026 19:30
…elease The `reassert_flux_ghcr_auth` step ran under `always()`, which also fires after a job is cancelled. It acquires the `ghcr-auth-refresh` Lease, whose design deliberately has no automatic expiry takeover: Talos machine-config writes expose no fencing token, so a non-empty holder always requires explicit human recovery. A cancelled job is force-killed once the runner's post-cancellation grace expires, so the EXIT trap's release never lands and the Lease keeps a dead holder — after which every later deploy fails to acquire it. Merge-queue evictions cancel this job routinely, so the state is reachable in normal operation. Run the step on failure but not on cancellation; the next deploy's staging step performs the same reassertion with a full time budget. Adding a signal trap is not an alternative: a trapped TERM makes bash defer the handler until the in-flight child returns, which under a kill window is strictly worse than the current immediate EXIT-trap path.
devantler
force-pushed
the
claude/ghcr-lease-cancellation-recovery
branch
from
August 10, 2026 08:06
6575ec2 to
da69fd1
Compare
Code Coverage OverviewLanguages: Go Go / code-coverage/goThe overall coverage in commit da69fd1 in the Show a code coverage summary of the most covered files.
|
devantler
added this pull request to the merge queue
Aug 10, 2026
Any commits made after this event will not be merged.
Contributor
Author
Heads-up: this PR's head was refreshed by an autonomous run that misclassified it as routine-owned.
No content was changed and no conflict arose. Two consequences worth knowing:
No further action will be taken on this PR by the scheduled engineer. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Prod CD has been failing every run with:
A merge-queue deploy was evicted (cancelled) on 2026-08-09. Its post-cancellation cleanup step still started, took the deploy lease, and was then force-killed when the runner's grace window ran out — leaving the lease held by a dead process. That lease deliberately has no automatic takeover, so every deploy after it fails, permanently, until a human clears it. Merge-queue evictions are routine here, so this is reachable in normal operation rather than a freak event.
What
The cleanup step now runs on failure but not on cancellation. Its real purpose — repairing auth when
cluster updatefails part-way — is unchanged; only the cancelled path is dropped, and that path could never finish inside the grace window anyway. The next deploy performs the same reassertion with a full time budget.Also adds a contract test asserting that no step which takes the lease is reachable after cancellation, so this cannot regress into any future step. The sibling step in
dr-rebuild.yamlalready used this condition — this brings the deploy action in line with it.🤖 Generated with Claude Code