Skip to content

gate:matrix: matrix-run.json's section9_floor.meets_floor can disagree with report.json's #95

Description

@myselfsiddharth

Follow-up from review of #92.

run-matrix.ts writes a section9_floor block into out/matrix-run.json computed inline as:

meets_floor: runs.length >= 42 && runs.length * program.steps.length >= 400,

This assumes every completed run executed every step in the program. That's not guaranteed: ReplayRunner.run() sets steps_total = program.steps.length unconditionally, but the actual emitted step rows (stepResults) can be shorter, because the run loop breaks after a hard failure (if (!SUCCESS_OUTCOMES.has(final.outcome)) break; in src/runner/replay.ts).

src/metrics/aggregate.ts's section9SampleFloor() computes the same concept correctly from the actual NDJSON rows (dedupeLatestSteps(filterSteps(rows)).length), and that's what gate:report's report.json uses.

So once any run in the matrix fails partway through — which is the scenario this whole harness exists to observe — matrix-run.json.section9_floor.meets_floor can read true while report.json.sample.meets_floor (built from the same NDJSON) reads false for the same measurement.

Fix: have run-matrix.ts call section9SampleFloor() on the actual persisted rows (e.g. readMetricNdjson(ndjsonPath) after the final persist()) instead of re-deriving the floor check by hand, so both artifacts agree by construction.

Not currently exercised by any test — the only program in the matrix today (the 2-step example bundle) hasn't shown a partial-run failure in measured runs, so runs.length * steps.length happens to equal the true step-execution count. Worth a regression test once the real gate task (#25) can fail mid-run.

Metadata

Metadata

Assignees

Labels

gatePRD section 9 gate measurement

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions