CI retries cost 3x wall clock and rescue almost nothing

[sbryngelson]

Split out of #1797, which touched adjacent code but deliberately left this alone — it is a behavioural change to the test harness rather than CI plumbing.

The measurement

Over 540 first-attempt job failures (2026-08-18..31), every retry mechanism in the CI path:

mechanism	cases retried	rescued by the retry
bench.py per-case (max_attempts = 2)	235	0
mfc.sh test --max-attempts 3	2,795 failed-test records	every one shows Attempts: 3
build.sh / retry_build (2 attempts)	210 Build failed after 2 attempts	0 — Build succeeded on attempt 2 appears zero times in 552 logs
retry_sbatch (3 attempts)	1 job ever reached attempt 2	81 correctly refused as non-transient
GitHub whole-run re-run	18 reaching a verdict	11 (61%)

Only the whole-run re-run helps, and only because hours pass and the node or queue situation changes in the meantime.

Why the in-process retries cannot help

handle_case in toolchain/mfc/test/test.py retries on any exception:

except Exception as exc:
    if nAttempts < max_attempts:
        continue

That includes golden-file tolerance mismatches — the single most common failure class in the window (88 of 540) — which are deterministic by construction. Re-running the same binary on the same input cannot produce a different comparison. The same applies to compile errors surfaced through the same path.

So the suite spends 3x the wall clock on the failures it is least able to fix, and the retry is also what makes a real failure take three times as long to surface.

Suggested direction

Retry only classes that are plausibly transient — execution failures, launcher/MPI faults, timeouts — and fail fast on deterministic ones (tolerance/golden mismatch, compile errors). handle_case already classifies the exception for its error_type field, so the information needed is present at the decision point; it just is not consulted before continue.

Worth pairing with a count of how often a retry does rescue a case, so the value is measurable rather than assumed.

Note

bench.py's 0-of-235 is the cleanest evidence, because there both the retry and its outcome are logged. The test-suite figure is one-sided by construction — only failures record Attempts: — but 2,795 records all at the maximum, with zero observed rescues anywhere else, makes a sizeable rescue rate implausible.

[sbryngelson]

Partly addressed in #1797 (e20de7c1) — the two parts that need no judgement call about which failures are transient:

Rescues are now measured. nPASS was incremented identically whether a case passed on attempt 1 or attempt 3, so a successful retry left no trace anywhere. That is the reason the evidence above is one-sided: failures record Attempts: N, successes recorded nothing, so the rescue rate could be argued about but not measured. A pass after a retry is now counted, logged as it happens, and reported in the run summary:

    10 passed
     2 failed
     3 recovered by a retry

One CI cycle across the matrix turns the open question in this issue into a number.

Retries no longer run through an abort. The retry loop never consulted abort_tests. The suite-wide abort fires when the failure rate says the environment is broken — a dead GPU, a bad node — which is exactly when every remaining attempt is certain to fail. Cases already in their retry loop spent the rest of their budget anyway, so the fail-fast path was slowed threefold by the retry it was trying to escape. The decision now lives in a pure should_retry(attempt, max_attempts, aborting) with unit tests.

Still open, and deliberately so: which classes are worth retrying at all. That is the substantive half of this issue, and it should be decided against the rescue numbers the above starts collecting rather than against the one-sided sample I opened with. Worth revisiting once a week or so of runs have accumulated — if the recovered count is near zero across the matrix, the case for dropping retries on deterministic classes makes itself.

[sbryngelson]

Correcting the framing in this issue — I overstated the cost, and the measurement changes the conclusion.

"Retries cost 3x the wall clock" is true per failing test, but as written above it reads as 3x overall. I never checked. Measuring it across the same window:

2,796 failed-test records x 2 extra attempts x ~29s mean test duration
  = ~45 h, against 730 h of total first-attempt waste   ->  ~6%

And that 6% is not spread evenly:

median failing job:      3 failed tests   (~3 min of retry overhead in a ~100 min job)
top 20 jobs:             56% of all retry cost
worst single job:      108 failed tests

The concentration is in mass-failure jobs — bad nodes, broken environments — which is exactly what the 30% failure-rate abort exists to stop. And the abort was being defeated by the retry, because the retry loop never consulted abort_tests. That is fixed (#1797, e20de7c1), which recovers the majority of the recoverable cost.

What remains is a policy question governing roughly three minutes per failing job. That is worth doing eventually, but not worth the ~120 lines of instrumentation and harvesting I initially added for it — I have reverted that (dfc71f88) rather than leave measurement apparatus in the tree for a question this size.

Still worth doing, at low priority: skip retries for deterministic failure classes. classify_error() now exists and is used at the retry decision point, so it is a small change when someone wants it. The plain rescued count remains in the summary, so if retries ever do start rescuing things, it will be visible without any extra machinery.

Leaving this open at reduced scope rather than closing it.