fix(release): break the stuck-tag deadlock in the nightly release guard#5593
Merged
Conversation
A tag whose Release run died before the publish gate (v0.21.0: Build Server (windows) hit the old 35-minute timeout and was cancelled) deadlocked the nightly permanently. Two independent defects compounded: 1. Signal mismatch. The version step derives next_minor from the highest PUBLISHED GitHub Release, but the existing-tag guard keyed off release-RUN conclusions. When those disagree the nightly jams: version computation keeps recomputing 0.21.0 (no published release), the tag already exists, and the guard hard-exits on the stale cancelled run. Nothing in the nightly could ever clear that. Key the guard off the published release instead -- the same fact the version step already uses -- so the two cannot disagree, and a tag with no published release self-heals into a recovery dispatch. 2. Recovery could never carry a fix. dispatch-recovery ran `gh workflow run release.yml --ref "$TAG"`, which EXECUTES the release.yml frozen in the tag's tree -- i.e. the same broken workflow that just failed (35m timeout). A CI fix landed on main could not reach a stuck tag. Dispatch from main instead; `-f tag=` still pins the built SOURCE to the tag SHA via resolve-release-ref, so this changes CI config only, never released code. Dropping the guard's "already in progress" branch (it matched runs by head_sha == TAG_SHA, which a main-ref dispatch no longer satisfies) means a tag-push and a recovery dispatch could in principle overlap, so release.yml gains a per-tag concurrency group. cancel-in-progress is false: a half-cancelled release is precisely what leaves a tag stuck.
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
matthewevans
enabled auto-merge
July 11, 2026 17:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A tag whose Release run died before the publish gate (v0.21.0: Build Server
(windows) hit the old 35-minute timeout and was cancelled) deadlocked the
nightly permanently. Two independent defects compounded:
Signal mismatch. The version step derives next_minor from the highest
PUBLISHED GitHub Release, but the existing-tag guard keyed off release-RUN
conclusions. When those disagree the nightly jams: version computation keeps
recomputing 0.21.0 (no published release), the tag already exists, and the
guard hard-exits on the stale cancelled run. Nothing in the nightly could
ever clear that. Key the guard off the published release instead -- the same
fact the version step already uses -- so the two cannot disagree, and a tag
with no published release self-heals into a recovery dispatch.
Recovery could never carry a fix. dispatch-recovery ran
gh workflow run release.yml --ref "$TAG", which EXECUTES the release.ymlfrozen in the tag's tree -- i.e. the same broken workflow that just failed
(35m timeout). A CI fix landed on main could not reach a stuck tag. Dispatch
from main instead;
-f tag=still pins the built SOURCE to the tag SHA viaresolve-release-ref, so this changes CI config only, never released code.
Dropping the guard's "already in progress" branch (it matched runs by
head_sha == TAG_SHA, which a main-ref dispatch no longer satisfies) means a
tag-push and a recovery dispatch could in principle overlap, so release.yml
gains a per-tag concurrency group. cancel-in-progress is false: a half-cancelled
release is precisely what leaves a tag stuck.