Skip to content

fix(ci): make Release single-flight and idempotent to end double-trigger 422s#491

Merged
joshua-temple merged 1 commit into
mainfrom
fix/release-double-trigger
Jul 6, 2026
Merged

fix(ci): make Release single-flight and idempotent to end double-trigger 422s#491
joshua-temple merged 1 commit into
mainfrom
fix/release-double-trigger

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

The Release workflow is triggerable three ways (push: tags: v*, release:, and workflow_dispatch) with no concurrency group. The redundancy is intentional (#86): orchestrate, promote, and auto-promote each dispatch Release explicitly because API-created and CI-skipped tags do not reliably emit push or release events.

When a candidate tag is cut by a real git push (which does emit a push event) and the dispatcher also fires its explicit dispatch, both trigger Release at the same instant. Two parallel runs race GoReleaser; the loser hits 422 already_exists, leaving a red Release run and a stray draft every cycle. This happened for v0.9.1-rc.1 (runs 28800047479 push/success and 28800046933 dispatch/failure, identical createdAt).

Fix

Make the duplicate harmless rather than removing a trigger (the fallback is still required for the inverse failure mode where the push event is suppressed):

  • Idempotency guard (primary). A pre-check at the head of the release job skips the build and finishes green when a live, non-draft release already carries its checksums.txt manifest for the tag. A belt after GoReleaser (run under continue-on-error) converts a GoReleaser failure into success only when the tag is already published, and fails the job on any other error. The decision core is .github/scripts/release-idempotency.sh, unit tested in internal/release/release_idempotency_test.go (existing-release-with-checksums to skip, draft/no-checksums/absent to build, and the 422-tolerance predicate).
  • Global single-flight group. concurrency: { group: release, cancel-in-progress: false }, a constant group rather than one keyed on the tag, enforces the hard invariant that no two releases are ever in flight together and serialises the double-trigger.

Cancel-pending safety analysis

A shared group with cancel-in-progress: false still cancels all-but-the-latest pending run. Trace for a legitimate final publish:

  • The final publish run is single, not doubled: auto-promote pushes the final tag with the default GITHUB_TOKEN, whose pushes do not trigger workflows, then dispatches Release once. It is also temporally far from the rc build runs (a full fleet gate, tens of minutes, sits between them), so it enters the release group alone and is not pending behind anything in the normal flow.
  • The rc build double-trigger produces exactly two runs (push + dispatch). One runs, one queues; a pending run is only cancelled when a third trigger arrives, which the serialized pipeline does not produce.
  • Even in the worst case where a pending publish is superseded, it is never silently dropped: the release job is idempotent, and every dispatcher watches its Release run (auto-promote's watch step surfaces a cancelled run as a failed promote, not a silent skip) and can redispatch. A redispatch re-runs the idempotent job, which publishes if absent or no-ops if already published.

Conclusion: the global group is safe; a legitimate final publish is either not pending, or is recoverable by the idempotent, watched, redispatchable path. No design change away from the global group is needed.

Per-caller trace

  • rc-tag push (build rc): serialised with the dispatch; winner publishes, loser idem-skips green.
  • orchestrate explicit dispatch (rc build): the loser of the pair; idem-skips or the 422 belt absorbs a simultaneous publish.
  • auto-promote dispatch (final publish): single run, temporally isolated; publishes normally.
  • promote.yaml dispatch: same single-flight and idempotent semantics.
  • hotfix / manual tag push: enters the release group; publishes or idem-skips if already published.
  • release: event fallback: same guards apply; a duplicate no-ops.

Verification

  • go build ./..., go test ./... (2448 passed), golangci-lint run ./... (no issues), go vet clean.
  • actionlint .github/workflows/release.yaml clean; shellcheck clean on the new script.
  • New regression tests fail-shape without the script and pass with it.
  • Docs updated (release-orchestration.md, new "Release runs once per tag" section). Local Node is v20.20.2 (below 22.12); CI builds the docs site. The addition is plain prose and will not break the build.

Residual risk

The 422 belt runs GoReleaser under continue-on-error, so a GoReleaser failure is only surfaced by the follow-up confirm step. That step fails the job on any failure other than an already-published tag, so real failures still red the run.

…ger 422s

The Release workflow is triggered redundantly on purpose: a candidate tag
push emits a real event while orchestrate, promote, and auto-promote also
dispatch it explicitly, because API-created and CI-skipped tags do not emit
push or release events (#86). A pushed tag then starts two runs at once, the
loser hits GoReleaser 422 already_exists, and each cycle leaves a red run and
a stray draft.

Add a constant single-flight concurrency group so duplicate runs serialise
instead of racing, and an idempotency guard that skips the build and finishes
green when the tag is already published with its checksum manifest. A belt
after GoReleaser treats a 422 as success only when the tag is already
published. The decision core lives in a script and is unit tested.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit 40391ff into main Jul 6, 2026
21 checks passed
@joshua-temple
joshua-temple deleted the fix/release-double-trigger branch July 6, 2026 17:42
joshua-temple added a commit that referenced this pull request Jul 6, 2026
…eneration

The tag-only/non-triggering-tag-create behavior is cascade's own release
plumbing (cascade self-publishes via GoReleaser), not a feature every
downstream manifest should see. Move it off the public manifest surface.

- remove the public release.tag_only manifest field entirely: config schema,
  validation, and docs are back to plain generation; downstream manifest
  generation is byte-identical to before this change
- add an own-repo generation mode instead, mirroring the reconcile
  companion's WithOwnRepo: NewGenerator gains a variadic GeneratorOption tail
  and WithOwnRepoRelease(), which is reachable only from a new --own-repo
  flag on generate-workflow and verify (never from config)
- the manage-release composite action template only adds the tag_only input
  (and its env/CLI forwarding) when generated in own-repo mode; plain
  generation carries no trace of it
- regenerate cascade's own orchestrate.yaml and manage-release action with
  --own-repo; leave promote.yaml untouched (cascade's real final publish is
  the hand-authored auto-promote.yaml; promote.yaml is inert for cascade)
- wire --own-repo into the PR workflow-drift check and its regenerate hints
  so cascade's own-repo generated files are drift-locked correctly
- rework e2e scenario 46 to exercise generate-workflow --own-repo instead of
  a manifest field, and add scenario 47 as the paired plain-generation
  regression guard
- keep the release.go/command.go runtime mechanics (tag-only create()
  short-circuit, --tag-only CLI flag, non-triggering token, #491 backstop)
  unchanged; only the scoping vehicle changed

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple added a commit that referenced this pull request Jul 6, 2026
…496)

* fix(release): one Release run per tag and no orphan draft on rc cut

Cascade's own rc cut and final publish each drove two Release runs per tag
(the state-token tag push fired push: tags while orchestrate/promote also
dispatched Release) and the rc cut left an orphaned draft that GoReleaser
duplicated. Both are now single-fire, mirroring the auto-promote pattern.

- add release.tag_only (schema, validation, manage-release --tag-only): cut
  the tag only, skip the draft-release POST, and create the tag with the
  non-triggering GITHUB_TOKEN so the tag push does not fire the release
  workflow alongside the explicit dispatch
- orchestrate rc cut and promote publish emit tag_only + GITHUB_TOKEN for the
  tag-create step in tag-only mode, keeping the explicit dispatch on the PAT so
  the Release run still cascades to fleet-e2e via workflow_run
- scope tag_only to cascade's own manifest; downstream generated output is
  byte-identical (regression-guarded), so a repo without GoReleaser keeps its
  draft-then-publish flow
- keep push: tags for the hotfix and human ad-hoc tag paths; dispatch is
  canonical for machine paths
- retain the release concurrency group, release-idempotency.sh, and benign-422
  belt as a defensive backstop
- regenerate cascade's orchestrate/promote/manage-release; set tag_only in the
  manifest; e2e scenario 46; unit and regression tests; docs

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>

* fix(release): scope tag-only release plumbing to cascade's own-repo generation

The tag-only/non-triggering-tag-create behavior is cascade's own release
plumbing (cascade self-publishes via GoReleaser), not a feature every
downstream manifest should see. Move it off the public manifest surface.

- remove the public release.tag_only manifest field entirely: config schema,
  validation, and docs are back to plain generation; downstream manifest
  generation is byte-identical to before this change
- add an own-repo generation mode instead, mirroring the reconcile
  companion's WithOwnRepo: NewGenerator gains a variadic GeneratorOption tail
  and WithOwnRepoRelease(), which is reachable only from a new --own-repo
  flag on generate-workflow and verify (never from config)
- the manage-release composite action template only adds the tag_only input
  (and its env/CLI forwarding) when generated in own-repo mode; plain
  generation carries no trace of it
- regenerate cascade's own orchestrate.yaml and manage-release action with
  --own-repo; leave promote.yaml untouched (cascade's real final publish is
  the hand-authored auto-promote.yaml; promote.yaml is inert for cascade)
- wire --own-repo into the PR workflow-drift check and its regenerate hints
  so cascade's own-repo generated files are drift-locked correctly
- rework e2e scenario 46 to exercise generate-workflow --own-repo instead of
  a manifest field, and add scenario 47 as the paired plain-generation
  regression guard
- keep the release.go/command.go runtime mechanics (tag-only create()
  short-circuit, --tag-only CLI flag, non-triggering token, #491 backstop)
  unchanged; only the scoping vehicle changed

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>

---------

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant