Skip to content

test_op_parity is RED on main: RunGoldenPass indexes m["op"] on every manifest, and the MUSIC3 oracle capture has no such key #755

Description

@localai-bot

The defect

test_op_parity is RED on main. RunGoldenPass walks every manifest.json
under tests/parity/goldens/ and indexes m["op"] unconditionally, but
tests/parity/goldens/minimax_music3_oracle/manifest.json has no op key —
its keys are captured_on, checkpoint, environment, generated_by, issue, model, oracle, request.

Verified independently on current main:

json.load(open('tests/parity/goldens/minimax_music3_oracle/manifest.json'))
# has op key: False

It arrived with 34dc57876 (MODEL-MUSIC-MUSIC3, #672 / #708) — an oracle
capture manifest
, which is a different kind of artifact from the per-op parity
goldens that directory otherwise holds.

Why it matters more than one red test

The goldens tree is a shared surface with an implicit schema, and nothing
declares it. Any row that captures oracle evidence into
tests/parity/goldens/<name>/manifest.json — a reasonable place to put it —
silently breaks a test belonging to a different row. It has already happened
once; the next campaign to record an oracle capture there will hit it again.

It also cost time on an unrelated row: the NemotronH loader's ctest came back
453 of 454 and the one failure had to be traced and excluded before its own
numbers could be trusted. That is the tax a shared surface with an undeclared
schema charges every passer-by, which AGENTS.md calls out directly — "if a
checker requires every change to touch a shared file, that is the defect"
, and
the same logic applies to a checker that requires every change to conform to an
unwritten schema
in a shared directory.

Options, none of them chosen here

  1. Give the manifest an op key — smallest diff, but it makes an oracle
    capture pretend to be an op golden.
  2. Skip manifests without op in RunGoldenPass — but a silent skip is how
    a golden stops being gated, and this repo has been bitten by exactly that.
    If taken, the skip must be loud and counted.
  3. Separate the trees — oracle captures under their own root, so the parity
    walker's input set is closed by construction. Most work, least recurrence.

Option 3 is the one that stops it happening a third time; option 2 without a
loud skip is the one that looks fixed and is not.

Done when

test_op_parity is green on main, and a manifest that does not match the
walker's expected shape is either impossible to place there or fails loudly
and by name
rather than through an unhandled key lookup.

Found while gating the NemotronH weight loader (#517); the failure is on the
base, not that branch — its diff touches nothing under tests/parity/ or
src/vt/.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions