Skip to content

feat(guard): require a bounded timeout-minutes on every required-context job (#398) - #411

Merged
avrabe merged 1 commit into
mainfrom
feat/required-context-timeouts-398
Aug 8, 2026
Merged

feat(guard): require a bounded timeout-minutes on every required-context job (#398)#411
avrabe merged 1 commit into
mainfrom
feat/required-context-timeouts-398

Conversation

@avrabe

@avrabe avrabe commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Closes #398.

The problem

A required status check that hangs is not a wrong-green — it is an indefinite merge block with no signal. A wedged job leaves its context at Expected — waiting for status to be reported, which is byte-identical to "still running", and a job with no timeout-minutes inherits GitHub's 360-minute default before anything cancels it. For up to six hours the PR is unmergeable for a reason nobody can see, the job holds a self-hosted runner, and — branch protection is strict = true — every other open PR queues behind it. Detect changed paths is the sharpest case: it gates ten of the other jobs, so wedging it stalls the entire suite.

Measured against the live required set: 15 of the 18 required contexts declared no timeout-minutes. The 3 that did: Mutation Testing (240), Verification Gate (60), Lean typecheck (90).

What this does

1. A guard — tools/check_job_timeouts.py. A sibling of check_required_contexts.py that reuses its restricted, PyYAML---cross-checked workflow reader rather than copying it. Every GATEABLE job — exactly the set that gate proves must be required — must declare a timeout-minutes that is a static integer in the open interval (0, 360):

  • missing → the defect (inherits the 360 default);
  • ${{ … }} template → not a static bound a reader can evaluate;
  • <= 0 → not a number of minutes;
  • >= 360 → no tighter than the default it replaces, so it buys nothing.

Advisory (continue-on-error) and undeliverable (path-scoped) jobs are out of scope — they can't block a merge. The scan fails closed: zero gateable jobs found, or a glob that misses .yaml, is a blind read, not a PASS.

The check_required_contexts.py reader is extended (additively) to capture timeout-minutes; --self-test and --cross-check on it still pass.

2. The 15 missing values, sized to a few multiples of each job's observed p95 (cold-cache worst case), cited per-job in the workflow comments:

jobs timeout
Detect changed paths (×2), Format, Cargo Deny, Supply Chain 5
Security Audit (RustSec) 15
Test, Proptest, Bench compile smoke, Code Coverage, Miri, Fuzz smoke 30
Clippy, Rivet validate 45
Codegen compile oracle 60

Durations were read from recent CI run history (2 main runs + 1 PR run). This does not try to machine-check that the values are well-sized — that's a human judgement; the gate only draws the line that isn't one: an unbounded (or uselessly-bounded) required job must not ship.

Oracle (executed, non-vacuous)

tools/check_job_timeouts.py --self-test — a 9-row decision table plus two production-corpus assertions — runs before the gate, as a step of the rivet-validate job (same steps-of-this-job rationale as the sibling: a new job would itself be gateable and need admin-token branch-protection sync).

  • Distinct inputs → distinct verdicts: valid → 0, missing/templated/zero/non-integer/>=360 → 1, 359 → 0, advisory/undeliverable with no timeout → 0.
  • Fails closed on a blind scan (zero gateable / glob misses .yaml).
  • Demonstrated live: against the pre-change tree the check exits 1 naming all 15 unbounded contexts; against this branch it exits 0 with "18 gateable job(s)".

An independent clean-room pass re-derived the 15/18 numbers, proved non-vacuity by sabotaging validate_timeout (self-test then fails), and checked the PROVEN-vs-ASSUMED honesty — no over-claim found.

Honest scoping

PROVEN: every job backing a required context declares a static bound in (0, 360), and the gate fails closed on a missing/templated/degenerate bound and on a blind scan.
ASSUMED (not proven here): that the chosen minute values are correctly sized; that GitHub actually cancels at the declared minute (runtime, not exercised); that the GATEABLE set equals the branch-protection-enforced set (the admin-token gap check_required_contexts.py states and this inherits).
NOT claimed: that advisory jobs are bounded — a hung continue-on-error job wastes a runner but can't wedge the merge gate; left to a successor.

Tracked as REQ-GUARD-CONTEXT-TIMEOUT-001 / TEST-GUARD-CONTEXT-TIMEOUT (real steps + satisfies/verifies links). rivet validate: 0 broken cross-refs.

🤖 Generated with Claude Code


Generated by Claude Code

…ext job (#398)

A required status check that hangs is an indefinite merge block with no
signal: the context sits at "Expected — waiting for status" (identical to
"still running"), and a job with no timeout-minutes inherits GitHub's
360-minute default before anything cancels it — holding a self-hosted runner
and, with strict branch protection, queuing every other PR. 15 of the 18
required contexts declared no timeout-minutes; Detect changed paths, which
gates ten other jobs, was the sharpest case.

Add tools/check_job_timeouts.py, a sibling of check_required_contexts.py that
reuses its restricted, PyYAML-cross-checked workflow reader: every GATEABLE
job (exactly the set that gate proves must be required) must declare a static
timeout-minutes in (0, 360). Missing, templated, <=0, or >=360 all fail; the
scan fails closed on zero gateable jobs or a glob that misses .yaml. A 9-row
self-test flips a verdict each way (missing->1, valid->0, advisory/undeliverable
no-timeout->0) and runs before the gate, in the rivet-validate job.

Give the 15 unbounded jobs a timeout sized to a few multiples of their observed
p95 (cold-cache worst case), cited per job: classifiers/fmt/deny/vet at 5,
audit 15, test/proptest/bench/coverage/miri/fuzz 30, clippy/rivet-validate 45,
codegen-oracle 60. Extend check_required_contexts.py's reader to capture
timeout-minutes (additive; cross-check unaffected).

Track as REQ-GUARD-CONTEXT-TIMEOUT-001 / TEST-GUARD-CONTEXT-TIMEOUT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Rivet verification gate

20/20 passed

count
Passed 20
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.

@avrabe
avrabe merged commit df4a64f into main Aug 8, 2026
21 checks passed
@avrabe
avrabe deleted the feat/required-context-timeouts-398 branch August 8, 2026 14:42
@avrabe avrabe mentioned this pull request Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

15 of 18 required contexts declare no timeout-minutes — a hung merge gate is a 6-hour blind wait

2 participants