Skip to content

fix(release): break the stuck-tag deadlock in the nightly release guard#5593

Merged
matthewevans merged 1 commit into
mainfrom
ship/release-stuck-tag-deadlock
Jul 11, 2026
Merged

fix(release): break the stuck-tag deadlock in the nightly release guard#5593
matthewevans merged 1 commit into
mainfrom
ship/release-stuck-tag-deadlock

Conversation

@matthewevans

Copy link
Copy Markdown
Member

A tag whose Release run died before the publish gate (v0.21.0: Build Server
(windows) hit the old 35-minute timeout and was cancelled) deadlocked the
nightly permanently. Two independent defects compounded:

  1. Signal mismatch. The version step derives next_minor from the highest
    PUBLISHED GitHub Release, but the existing-tag guard keyed off release-RUN
    conclusions. When those disagree the nightly jams: version computation keeps
    recomputing 0.21.0 (no published release), the tag already exists, and the
    guard hard-exits on the stale cancelled run. Nothing in the nightly could
    ever clear that. Key the guard off the published release instead -- the same
    fact the version step already uses -- so the two cannot disagree, and a tag
    with no published release self-heals into a recovery dispatch.

  2. Recovery could never carry a fix. dispatch-recovery ran
    gh workflow run release.yml --ref "$TAG", which EXECUTES the release.yml
    frozen in the tag's tree -- i.e. the same broken workflow that just failed
    (35m timeout). A CI fix landed on main could not reach a stuck tag. Dispatch
    from main instead; -f tag= still pins the built SOURCE to the tag SHA via
    resolve-release-ref, so this changes CI config only, never released code.

Dropping the guard's "already in progress" branch (it matched runs by
head_sha == TAG_SHA, which a main-ref dispatch no longer satisfies) means a
tag-push and a recovery dispatch could in principle overlap, so release.yml
gains a per-tag concurrency group. cancel-in-progress is false: a half-cancelled
release is precisely what leaves a tag stuck.

A tag whose Release run died before the publish gate (v0.21.0: Build Server
(windows) hit the old 35-minute timeout and was cancelled) deadlocked the
nightly permanently. Two independent defects compounded:

1. Signal mismatch. The version step derives next_minor from the highest
   PUBLISHED GitHub Release, but the existing-tag guard keyed off release-RUN
   conclusions. When those disagree the nightly jams: version computation keeps
   recomputing 0.21.0 (no published release), the tag already exists, and the
   guard hard-exits on the stale cancelled run. Nothing in the nightly could
   ever clear that. Key the guard off the published release instead -- the same
   fact the version step already uses -- so the two cannot disagree, and a tag
   with no published release self-heals into a recovery dispatch.

2. Recovery could never carry a fix. dispatch-recovery ran
   `gh workflow run release.yml --ref "$TAG"`, which EXECUTES the release.yml
   frozen in the tag's tree -- i.e. the same broken workflow that just failed
   (35m timeout). A CI fix landed on main could not reach a stuck tag. Dispatch
   from main instead; `-f tag=` still pins the built SOURCE to the tag SHA via
   resolve-release-ref, so this changes CI config only, never released code.

Dropping the guard's "already in progress" branch (it matched runs by
head_sha == TAG_SHA, which a main-ref dispatch no longer satisfies) means a
tag-push and a recovery dispatch could in principle overlap, so release.yml
gains a per-tag concurrency group. cancel-in-progress is false: a half-cancelled
release is precisely what leaves a tag stuck.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@matthewevans
matthewevans enabled auto-merge July 11, 2026 17:07
@matthewevans
matthewevans added this pull request to the merge queue Jul 11, 2026
Merged via the queue into main with commit 66665ff Jul 11, 2026
13 checks passed
@matthewevans
matthewevans deleted the ship/release-stuck-tag-deadlock branch July 11, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant