Follow-up from review of #92.
run-matrix.ts writes a section9_floor block into out/matrix-run.json computed inline as:
meets_floor: runs.length >= 42 && runs.length * program.steps.length >= 400,
This assumes every completed run executed every step in the program. That's not guaranteed: ReplayRunner.run() sets steps_total = program.steps.length unconditionally, but the actual emitted step rows (stepResults) can be shorter, because the run loop breaks after a hard failure (if (!SUCCESS_OUTCOMES.has(final.outcome)) break; in src/runner/replay.ts).
src/metrics/aggregate.ts's section9SampleFloor() computes the same concept correctly from the actual NDJSON rows (dedupeLatestSteps(filterSteps(rows)).length), and that's what gate:report's report.json uses.
So once any run in the matrix fails partway through — which is the scenario this whole harness exists to observe — matrix-run.json.section9_floor.meets_floor can read true while report.json.sample.meets_floor (built from the same NDJSON) reads false for the same measurement.
Fix: have run-matrix.ts call section9SampleFloor() on the actual persisted rows (e.g. readMetricNdjson(ndjsonPath) after the final persist()) instead of re-deriving the floor check by hand, so both artifacts agree by construction.
Not currently exercised by any test — the only program in the matrix today (the 2-step example bundle) hasn't shown a partial-run failure in measured runs, so runs.length * steps.length happens to equal the true step-execution count. Worth a regression test once the real gate task (#25) can fail mid-run.
Follow-up from review of #92.
run-matrix.tswrites asection9_floorblock intoout/matrix-run.jsoncomputed inline as:This assumes every completed run executed every step in the program. That's not guaranteed:
ReplayRunner.run()setssteps_total = program.steps.lengthunconditionally, but the actual emitted step rows (stepResults) can be shorter, because the run loop breaks after a hard failure (if (!SUCCESS_OUTCOMES.has(final.outcome)) break;insrc/runner/replay.ts).src/metrics/aggregate.ts'ssection9SampleFloor()computes the same concept correctly from the actual NDJSON rows (dedupeLatestSteps(filterSteps(rows)).length), and that's whatgate:report'sreport.jsonuses.So once any run in the matrix fails partway through — which is the scenario this whole harness exists to observe —
matrix-run.json.section9_floor.meets_floorcan readtruewhilereport.json.sample.meets_floor(built from the same NDJSON) readsfalsefor the same measurement.Fix: have
run-matrix.tscallsection9SampleFloor()on the actual persisted rows (e.g.readMetricNdjson(ndjsonPath)after the finalpersist()) instead of re-deriving the floor check by hand, so both artifacts agree by construction.Not currently exercised by any test — the only program in the matrix today (the 2-step example bundle) hasn't shown a partial-run failure in measured runs, so
runs.length * steps.lengthhappens to equal the true step-execution count. Worth a regression test once the real gate task (#25) can fail mid-run.