Context
ReplayRunner caps how many times it will repair (maxRepairsPerRun, default 2 —
src/runner/replay.ts:76) but nothing caps elapsed time. A run's worst case is therefore
unbounded in the dimension a user actually feels.
Rough shape of the worst case with today's constants: each failing step spends its full
assertion timeout (DEFAULT_ASSERTION_TIMEOUT_MS, 5000 ms) before it is even classified as a
failure, then repair proposes and the step is retried, spending it again. A 12-step task with
several stale locators can sit for a long time with no output and no upper bound — and once
#27 wires a real repair model, each proposal adds model latency on top.
This matters for the product claim, not just for tests. PRD §3 sells replay as "near-zero
tokens, near-zero latency", and §9's kill line is explicitly "mean repair cost ≥ 70% of
fresh-reasoning cost (tokens and wall-clock, measured, not estimated)". A run with no time
ceiling can blow that line without anything reporting that it did.
What to build
A per-run wall-clock budget with an honest outcome when it is exhausted.
The hard part is not the timer — it is reporting. contracts/metrics.schema.json
$defs.stepOutcome has no "we ran out of time" member, and the repo rule is that aggregates
report no_data rather than invent a value. So decide, and write it down:
- Does budget exhaustion map onto the existing
TIMEOUT / REPAIR_EXHAUSTED outcomes, or does
it need a new one? A new member is a contract change.
- Is a truncated run's step-validity a measurement at all, or must it be excluded from the
denominator? Counting the unreached steps as failures would invent a result; silently
dropping them would shrink the denominator without saying so. Either choice must be visible
in the emitted rows.
Constraints
- Do not weaken assertions to fit a budget. Cutting an assertion's timeout to make a run finish
changes what is being measured — see the timeout_ms note in docs/gate/compiler.md.
- No invented metrics (CONTRIBUTING rule 3).
- Default should be generous enough that it never fires on a healthy run; it is a guard rail,
not a scheduler.
How to test
npm run test # a run that would exceed the budget stops, and says so in the metric row
npm run ci
npm run test:canary
Found while bounding the runner's waits for a latency review; the unbounded networkidle was
fixed separately, this is the remaining unbounded dimension.
Context
ReplayRunnercaps how many times it will repair (maxRepairsPerRun, default 2 —src/runner/replay.ts:76) but nothing caps elapsed time. A run's worst case is thereforeunbounded in the dimension a user actually feels.
Rough shape of the worst case with today's constants: each failing step spends its full
assertion timeout (
DEFAULT_ASSERTION_TIMEOUT_MS, 5000 ms) before it is even classified as afailure, then repair proposes and the step is retried, spending it again. A 12-step task with
several stale locators can sit for a long time with no output and no upper bound — and once
#27 wires a real repair model, each proposal adds model latency on top.
This matters for the product claim, not just for tests. PRD §3 sells replay as "near-zero
tokens, near-zero latency", and §9's kill line is explicitly "mean repair cost ≥ 70% of
fresh-reasoning cost (tokens and wall-clock, measured, not estimated)". A run with no time
ceiling can blow that line without anything reporting that it did.
What to build
A per-run wall-clock budget with an honest outcome when it is exhausted.
The hard part is not the timer — it is reporting.
contracts/metrics.schema.json$defs.stepOutcomehas no "we ran out of time" member, and the repo rule is that aggregatesreport
no_datarather than invent a value. So decide, and write it down:TIMEOUT/REPAIR_EXHAUSTEDoutcomes, or doesit need a new one? A new member is a contract change.
denominator? Counting the unreached steps as failures would invent a result; silently
dropping them would shrink the denominator without saying so. Either choice must be visible
in the emitted rows.
Constraints
changes what is being measured — see the
timeout_msnote indocs/gate/compiler.md.not a scheduler.
How to test
Found while bounding the runner's waits for a latency review; the unbounded
networkidlewasfixed separately, this is the remaining unbounded dimension.