Skip to content

Replay has no wall-clock budget — only a repair count cap #84

Description

@myselfsiddharth

Context

ReplayRunner caps how many times it will repair (maxRepairsPerRun, default 2 —
src/runner/replay.ts:76) but nothing caps elapsed time. A run's worst case is therefore
unbounded in the dimension a user actually feels.

Rough shape of the worst case with today's constants: each failing step spends its full
assertion timeout (DEFAULT_ASSERTION_TIMEOUT_MS, 5000 ms) before it is even classified as a
failure, then repair proposes and the step is retried, spending it again. A 12-step task with
several stale locators can sit for a long time with no output and no upper bound — and once
#27 wires a real repair model, each proposal adds model latency on top.

This matters for the product claim, not just for tests. PRD §3 sells replay as "near-zero
tokens, near-zero latency"
, and §9's kill line is explicitly "mean repair cost ≥ 70% of
fresh-reasoning cost (tokens and wall-clock, measured, not estimated)"
. A run with no time
ceiling can blow that line without anything reporting that it did.

What to build

A per-run wall-clock budget with an honest outcome when it is exhausted.

The hard part is not the timer — it is reporting. contracts/metrics.schema.json
$defs.stepOutcome has no "we ran out of time" member, and the repo rule is that aggregates
report no_data rather than invent a value. So decide, and write it down:

  • Does budget exhaustion map onto the existing TIMEOUT / REPAIR_EXHAUSTED outcomes, or does
    it need a new one? A new member is a contract change.
  • Is a truncated run's step-validity a measurement at all, or must it be excluded from the
    denominator? Counting the unreached steps as failures would invent a result; silently
    dropping them would shrink the denominator without saying so. Either choice must be visible
    in the emitted rows.

Constraints

  • Do not weaken assertions to fit a budget. Cutting an assertion's timeout to make a run finish
    changes what is being measured — see the timeout_ms note in docs/gate/compiler.md.
  • No invented metrics (CONTRIBUTING rule 3).
  • Default should be generous enough that it never fires on a healthy run; it is a guard rail,
    not a scheduler.

How to test

npm run test        # a run that would exceed the budget stops, and says so in the metric row
npm run ci
npm run test:canary

Found while bounding the runner's waits for a latency review; the unbounded networkidle was
fixed separately, this is the remaining unbounded dimension.

Metadata

Metadata

Labels

area: runnerTouches runnerenhancementNew feature or requestsize/M<= 200 changed linestrack-1Technical gate / churn harness

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions