Skip to content

Fix CI double-build race by keying concurrency on commit SHA - #1265

Merged
mkarlesky merged 2 commits into
next_versionfrom
ci-sha-scoped-concurrency
Sep 6, 2026
Merged

Fix CI double-build race by keying concurrency on commit SHA#1265
mkarlesky merged 2 commits into
next_versionfrom
ci-sha-scoped-concurrency

Conversation

@mkarlesky

Copy link
Copy Markdown
Member

Summary

Investigated why the existing push-vs-PR CI dedup mechanism
(check-duplicate-push, PR #1246) works inconsistently — "most instances
produce double builds, some skip the branch-push build."

Root cause: check-duplicate-push is a one-shot, point-in-time poll. On
a push event it queries gh pr list --head <branch> --state open once,
near the start of the run, and decides then and there whether to skip.
Push and PR-creation are independent, asynchronously-fired GitHub events
with no ordering guarantee between them. In the ordinary "push a branch,
then separately open a PR from it" flow — the dominant real-world case —
the poll almost always runs before the PR exists, finds nothing, and lets
the full run proceed to completion. The later pull_request-triggered run
then duplicates it, with no way to detect or cancel the redundant push run
(the concurrency: block was deliberately keyed by event_name
specifically so push and PR runs for the same commit couldn't cancel each
other — itself a fix for an earlier bug, commit bf878a97, where
branch-only concurrency grouping let a new commit's push run wrongly cancel
an older commit's still-running PR check). Outright skips of the push
run only happen in the narrower case where the PR already existed before
the push (e.g. a follow-up commit to an already-open PR).

Fix

Key the concurrency group on the commit SHA instead of branch+event, and
remove the custom polling job entirely:

concurrency:
  group: ci-${{ github.event.pull_request.head.sha || github.sha }}
  cancel-in-progress: true

github.event.pull_request.head.sha is the PR branch's real head commit on
a pull_request event; github.sha is the pushed commit on a push
event. For the identical commit, both resolve to the same group key — so a
push run and a later pull_request run for that commit now land in the
same concurrency group, and GitHub's own cancel-in-progress cancels
whichever started first the instant the second one starts. This is
event-driven, not a fixed-point-in-time guess, so it correctly handles a PR
being opened while the push run's multi-job matrix (several minutes) is
still in flight — not just the first few seconds after push. Keying on the
commit rather than the branch avoids reintroducing bf878a97's bug, since
a different commit on the same branch gets its own group.

Residual gap: if the push run fully completes before the PR is ever
opened, there's nothing left to cancel and it still runs twice for that
commit — a much narrower window than today's near-universal race.

Alternatives considered (not adopted)

  • Retry/poll with backoff in the existing push-side check: narrows but
    doesn't eliminate the race, and adds latency to every push run.
  • Narrow the push: trigger to protected branches only, relying on
    pull_request alone for feature branches: would eliminate the problem
    for the common case, but is a workflow philosophy change (no CI feedback
    on a bare push to a new feature branch until a PR exists), not a plumbing
    fix — kept as a separate, independent option if wanted later.
  • PR-side "was this SHA already built via push?" reuse check: not racy
    in that direction (a PR can only reference an already-pushed commit), and
    would close the residual gap above, but adds custom-script complexity for
    a narrow edge case — flagged for later if the residual gap proves to
    matter in practice.

Verification

This PR is itself a live test of the fix (push already happened; opening
this PR now exercises the exact "push, then promptly open a PR" scenario
the old mechanism handled worst). actionlint confirms no new issues
introduced. Full verification plan (see plan file) also covers a follow-up
commit to this open PR and a fresh-branch/immediate-PR test.

🤖 Generated with Claude Code

mkarlesky and others added 2 commits September 5, 2026 09:49
check-duplicate-push (PR #1246) was a one-shot poll: on a push event it
queried `gh pr list` once, near the start of the run, for an open PR on
that branch. Push and PR-creation are independent, asynchronously-fired
GitHub events with no ordering guarantee between them, so in the
ordinary "push a branch, then separately open a PR from it" flow, the
poll almost always ran before the PR existed, found nothing, and let
the full run proceed -- only to have the later pull_request-triggered
run duplicate it. The concurrency block couldn't help either: it was
deliberately keyed by event_name specifically so a push run and a
pull_request run for the same commit could never cancel each other
(itself a fix for an earlier bug, commit bf878a9, where branch-only
grouping let a new commit's push run wrongly cancel an older commit's
still-running PR check).

Keying the concurrency group on the commit SHA instead removes the
race and the custom polling job entirely: a push run and a later
pull_request run for the identical commit now land in the same group,
so GitHub's own cancel-in-progress cancels whichever started first the
moment the second one starts -- event-driven, not a fixed-point-in-time
guess, and correct for the realistic case of a PR being opened while
the push run's multi-job matrix is still in flight. Keying on the
commit rather than the branch avoids reintroducing bf878a9's bug,
since a different commit on the same branch gets its own group.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A feature/topic branch's own push event used to also trigger this
workflow, so a push immediately followed by opening a PR from it fired
two runs for the identical commit. Push and pull_request are
independent, asynchronously-fired events with no ordering guarantee
between them, so no point-in-time check on either side -- including
the SHA-scoped concurrency cancellation just added -- can prevent the
redundant run from being created in the first place, only clean it up
after the fact.

push: is now scoped to master and next_version only. A feature branch
gets CI exactly once, via the PR opened from it, removing the
duplicate trigger at its source rather than deduping it afterward. The
concurrency fix stays in place as a backstop for the narrower cases
this doesn't cover (e.g. a PR opened from master/next_version itself).

Also skips the full matrix for a still-draft PR until it's marked
ready, via a guard on every downstream job -- pull_request fires for
draft PRs the same as ready ones, and this team doesn't have a reason
to spend the whole matrix on a PR its author hasn't asked for review
on yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mkarlesky
mkarlesky merged commit bb3fc0c into next_version Sep 6, 2026
20 checks passed
@mkarlesky
mkarlesky deleted the ci-sha-scoped-concurrency branch September 6, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant