Skip to content

fix: evaluate status checks per name and page beyond the first 100 - #40

Closed
mmalyska wants to merge 3 commits into
masterfrom
fix/silent-check-skipping
Closed

fix: evaluate status checks per name and page beyond the first 100#40
mmalyska wants to merge 3 commits into
masterfrom
fix/silent-check-skipping

Conversation

@mmalyska

Copy link
Copy Markdown

Closes #39.

Consumer repository names, check names and commit SHAs are deliberately generic below: this repository is public, the repositories that consume it are not. The concrete identifiers behind each measurement are available internally.

The measurements that shaped this

PR #33 widened checkSuites/checkRuns to 100 and #35 reverted it on cost grounds. Both were reasoning about the right risk with the wrong numbers, so I measured before touching anything (rateLimit(dryRun: true), against a large internal monorepo, at -c 25 — what the shared promote workflow passes):

query shape commits nodeCount cost
before (suites 20 × runs 25) 25 13,075 6
before 100 52,300 23
PR #33 (100 × 100) 25 252,575 26
PR #33 (100 × 100) 100 MAX_NODE_LIMIT_EXCEEDED
this PR 25 5,050 1
this PR 100 20,200 3
  • revert: roll checkSuites/checkRuns page sizes back to PR-#33 originals #35's figures (131 → 2,526 points) were nodeCount/100, not the cost GitHub charges. The real increase was 6 → 26, so ~4.3×, not ~20×.
  • The increase was still worth reverting: the token is a GitHub App installation token shared by every repository using the shared reusable promote workflow, so that budget is org-wide, not per-repo. Hitting limits is entirely consistent with the arithmetic.
  • What should have blocked fix: raise checkSuites/checkRuns page size to 100 #33 outright is the last row: at --commits-number 100, the CLI's own default, that shape is rejected before it runs.

This PR goes the other way — 1 point at -c 25, 6× cheaper than today — while seeing more than #33 did.

What was actually broken

Counting, and it is live. checksPassed was incremented per matching check and compared with == against the number of requested names, so finding a name twice pushed the count past the target and rejected a commit whose checks had passed. One consumer repository publishes its aggregate required check as both a classic commit status and a check run on the same commit. It promotes today only because the check run falls outside the fetched window:

classicMatches=1  checkRunMatches=0  checksPassed=1  numChecks=1  → promotes

Widen the window and that becomes 2 == 1 → false → promotion silently stops. Any coverage increase, #33's included, had to land together with the counting fix. Names now resolve to a state instead of a tally.

Coverage. Check runs now come from statusCheckRollup.contexts, which is flat and de-duplicated by GitHub, so ordering across suites stops deciding whether a check is seen. Dropping the nested checkRuns connection is where the cost win comes from. I checked whether the rollup could replace checkSuites entirely (CheckRun.checkSuite.workflowRun.workflow.name costs 0 extra nodes) — it cannot: on a commit with over 130 check suites it yields 4 of 6 SUCCESS workflow names, so both connections stay.

The 100-per-page ceiling. Both connections cap at 100 and real repositories exceed it. Remaining pages are fetched by TopUpChecks from the evaluation loop, not from hydration — that loop already stops at the first passing commit, so the usual case costs no extra requests. Proven by asymmetry: at --check-suites-number 5 the first page contains no workflow runs at all, yet a required check that resolves only via a workflow name still comes back ✔.

Two things I did not expect

Defect 2 in the issue has no functional impact. The loop reads Commit.status.contexts, which is a plain LIST in GitHub's schema — no first, never truncated. The contexts(first: $parentsNumber) connection is on statusCheckRollup and selected only totalCount, which is read nowhere. It was dead weight, not an under-count. It now has a real purpose and its own variable.

Defaults are 50, not 100. A 100/100 first page makes GitHub exceed its own execution limit on the heaviest repository and return 504 — cost is not the only ceiling. At 50/50 the new query fails in exactly the same places the old one does (on that repository both fail above -c 25 and both succeed at -c 25, which is what the shared workflow passes), so this is not a new limit. GetCommits now names the knobs to turn when it happens.

Diagnosability

The issue asked for not-found to read differently from found-but-failing. It does now:

UNSATISFIED STATUS CHECKS ("not found" means no status, check run or workflow of that name exists on the commit):
  a1b2c3d4  failed: some-check
  e5f6a7b8  not found: some-other-check

That second line is not hypothetical — one consumer requests a check name that exists nowhere on its commits. No truncation involved, the name is simply stale, and until now that was indistinguishable from a failing build.

Verification

  • First commit is characterization tests only, green against unmodified logic, asserting the defects as they were. The port flips exactly three assertions, each marked WAS A DEFECT, NOW FIXED; every other expectation is unchanged.
  • go build, go vet, go test ./..., gofmt -l, golangci-lint (v1.44.2, as CI) all clean. A go test ./... job is added — CI previously ran lint only.
  • Re-ran all 11 consumer repositories before and after (testdata/baseline/, always --dry-run and with an unsatisfiable expression, so nothing is ever promoted): 10 byte-identical verdicts across 244 commits, 1 differing only by a commit that landed between runs while agreeing on all 24 shared commits, 0 verdict changes.

The baseline harness is tracked but its repository list and recordings are gitignored — recorded output embeds commit messages, author names and branch names, and this repository is public.

Not changed

One consumer's required check currently concludes NEUTRAL on recent commits, and NEUTRAL does not satisfy a check. This PR does not change which conclusions count as passing.

No tag. The shared reusable workflow defaults to the latest image tag, so releasing reaches every repository at once.

Characterization tests only — no production code is touched, so these pass on
master as-is. They exist to make the follow-up refactor's behaviour changes
explicit rather than incidental.

github/manager_test.go covers hydrateCommits and PickFirstParentCommits with
synthetic fixtures, and deliberately asserts today's defects as the expected
result, each marked KNOWN DEFECT:

  - a name found both as a classic status and as a check run is counted twice,
    so `checksPassed == numChecks` rejects a commit whose check did succeed
  - duplicate successful check runs with the same name are likewise counted
    once each, and blocked by the same equality gate
  - a name absent everywhere is indistinguishable from one that failed

The follow-up flips exactly those expectations; any other assertion moving means
something broke.

Not covered by unit tests, and noted as such in the file: a check that falls
outside checkSuites(first: 20) / checkRuns(first: 25) and is therefore never
fetched. hydrateCommits only sees what the query returned, so no fixture can
express it. testdata/baseline/ covers that end to end instead — it records the
per-commit verdict for real repositories so a query regression is visible as a
changed verdict.

The baseline harness is tracked; its repository list and recordings are
gitignored, since recorded output embeds commit messages, author names and
branch names and this repository is public.
Three defects made a commit silently fail evaluation, and because the CLI exits 0
either way the symptom was a promotion that just stopped moving.

Consumer repository and check names are kept generic here: this repository is
public, the repositories that consume it are not.

Counting. checksPassed was incremented once per matching check and compared for
equality with the number of requested names, so a name found twice pushed the
count past the target and rejected a commit whose checks had passed. This is not
hypothetical: one consumer repository publishes its aggregate required check as
both a classic commit status and a check run, and only promotes today because the
check run happens to fall outside the fetched window. Any widening of that
window — including PR #33's — would have broken it. Names now resolve to a state
rather than a tally, so duplicates and cross-source overlaps are harmless.

Coverage. checkSuites(first: 20) with a nested checkRuns(first: 25) meant a
required check could sit outside the window and be indistinguishable from one
that failed. Check runs now come from statusCheckRollup.contexts, which is flat
and de-duplicated by GitHub, so ordering across suites stops mattering. Dropping
the nested connection also removes a commits × suites × runs multiplier: for 25
commits the query goes from 13,075 nodes to 5,050, and its rate-limit cost from
6 points to 1. That is the objection behind PR #35's revert, answered — the
matching token is a GitHub App installation token shared by every repository
using the shared reusable promote workflow, so this budget is not per-repo.

Both connections cap at 100 per page and real repositories exceed it (the
heaviest consumer has over 130 check suites on a single commit). Remaining pages
are fetched per commit by TopUpChecks, called from the evaluation loop rather
than from hydration: that loop already stops at the first commit that passes, so
the usual case costs no extra requests and no commit is ever judged on a partial
view.

Diagnosability. --verbose now lists, per commit and per name, which checks were
not satisfied and whether each was found-but-failing or not found at all. Those
need opposite fixes and previously looked identical.

--contexts-number and --check-suites-number size only the first page, so they
trade requests against page size instead of dropping data. They default to 50,
not 100: on the heaviest repository a 100/100 first page makes GitHub exceed its
own execution limit and return 504. At 50/50 the new query fails in exactly the
same places the old one did (both fail above -c 25 and succeed at or below it,
which is what the shared workflow passes), so this is not a new limit.
GetCommits now says which knobs to turn when that happens.

Verified against the 11 repositories that consume the shared workflow, using the
recorded baseline: 10 byte-identical verdicts, 1 differing only by a commit that
landed between runs while agreeing on all 24 shared commits, 0 verdict changes.
The three unit assertions that flip are marked in manager_test.go.
Comment thread .github/workflows/pull_requests_tests.yaml Fixed
CodeQL flagged the new job for running with the default GITHUB_TOKEN scope. It
only reads the code, so grant it exactly that.
@mmalyska

Copy link
Copy Markdown
Author

Superseded by #41, which is stacked on #37.

#37 was already open and fixes the same counting defect, with better coverage than this PR had: it also catches the false-pass direction (a surplus on one name masking a deficit on another), trims names so a list with a space after the separator is satisfiable, and works through the policy question on SKIPPED/NEUTRAL. My mistake for not checking open PRs before starting.

#41 keeps only the part #37 explicitly deferred — the GraphQL page caps — rebased on top of it.

@mmalyska mmalyska closed this Aug 10, 2026
@mmalyska
mmalyska deleted the fix/silent-check-skipping branch August 10, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardcoded GraphQL first: limits can silently skip commits (checkSuites: 20 vs 27 real suites), plus contexts uses parentsNumber

2 participants