The defect
test_op_parity is RED on main. RunGoldenPass walks every manifest.json
under tests/parity/goldens/ and indexes m["op"] unconditionally, but
tests/parity/goldens/minimax_music3_oracle/manifest.json has no op key —
its keys are captured_on, checkpoint, environment, generated_by, issue, model, oracle, request.
Verified independently on current main:
json.load(open('tests/parity/goldens/minimax_music3_oracle/manifest.json'))
# has op key: False
It arrived with 34dc57876 (MODEL-MUSIC-MUSIC3, #672 / #708) — an oracle
capture manifest, which is a different kind of artifact from the per-op parity
goldens that directory otherwise holds.
Why it matters more than one red test
The goldens tree is a shared surface with an implicit schema, and nothing
declares it. Any row that captures oracle evidence into
tests/parity/goldens/<name>/manifest.json — a reasonable place to put it —
silently breaks a test belonging to a different row. It has already happened
once; the next campaign to record an oracle capture there will hit it again.
It also cost time on an unrelated row: the NemotronH loader's ctest came back
453 of 454 and the one failure had to be traced and excluded before its own
numbers could be trusted. That is the tax a shared surface with an undeclared
schema charges every passer-by, which AGENTS.md calls out directly — "if a
checker requires every change to touch a shared file, that is the defect", and
the same logic applies to a checker that requires every change to conform to an
unwritten schema in a shared directory.
Options, none of them chosen here
- Give the manifest an
op key — smallest diff, but it makes an oracle
capture pretend to be an op golden.
- Skip manifests without
op in RunGoldenPass — but a silent skip is how
a golden stops being gated, and this repo has been bitten by exactly that.
If taken, the skip must be loud and counted.
- Separate the trees — oracle captures under their own root, so the parity
walker's input set is closed by construction. Most work, least recurrence.
Option 3 is the one that stops it happening a third time; option 2 without a
loud skip is the one that looks fixed and is not.
Done when
test_op_parity is green on main, and a manifest that does not match the
walker's expected shape is either impossible to place there or fails loudly
and by name rather than through an unhandled key lookup.
Found while gating the NemotronH weight loader (#517); the failure is on the
base, not that branch — its diff touches nothing under tests/parity/ or
src/vt/.
The defect
test_op_parityis RED onmain.RunGoldenPasswalks everymanifest.jsonunder
tests/parity/goldens/and indexesm["op"]unconditionally, buttests/parity/goldens/minimax_music3_oracle/manifest.jsonhas noopkey —its keys are
captured_on, checkpoint, environment, generated_by, issue, model, oracle, request.Verified independently on current
main:It arrived with
34dc57876(MODEL-MUSIC-MUSIC3, #672 / #708) — an oraclecapture manifest, which is a different kind of artifact from the per-op parity
goldens that directory otherwise holds.
Why it matters more than one red test
The goldens tree is a shared surface with an implicit schema, and nothing
declares it. Any row that captures oracle evidence into
tests/parity/goldens/<name>/manifest.json— a reasonable place to put it —silently breaks a test belonging to a different row. It has already happened
once; the next campaign to record an oracle capture there will hit it again.
It also cost time on an unrelated row: the NemotronH loader's
ctestcame back453 of 454 and the one failure had to be traced and excluded before its own
numbers could be trusted. That is the tax a shared surface with an undeclared
schema charges every passer-by, which
AGENTS.mdcalls out directly — "if achecker requires every change to touch a shared file, that is the defect", and
the same logic applies to a checker that requires every change to conform to an
unwritten schema in a shared directory.
Options, none of them chosen here
opkey — smallest diff, but it makes an oraclecapture pretend to be an op golden.
opinRunGoldenPass— but a silent skip is howa golden stops being gated, and this repo has been bitten by exactly that.
If taken, the skip must be loud and counted.
walker's input set is closed by construction. Most work, least recurrence.
Option 3 is the one that stops it happening a third time; option 2 without a
loud skip is the one that looks fixed and is not.
Done when
test_op_parityis green onmain, and a manifest that does not match thewalker's expected shape is either impossible to place there or fails loudly
and by name rather than through an unhandled key lookup.
Found while gating the NemotronH weight loader (#517); the failure is on the
base, not that branch — its diff touches nothing under
tests/parity/orsrc/vt/.