tooling: coverage-parse-diff reports spurious oracle_changed and silently skips cards due to nondeterministic array ordering

[matthewevans]

Summary

coverage-parse-diff can report a nonzero oracle_changed — silently skipping cards instead of diffing them — purely because of array-ordering nondeterminism, with no underlying parse change. A reviewer reading that field as signal is misled, and the skipped cards are excluded from the diff entirely.

Found while measuring a doc/test-only engine change whose card-data.json was byte-identical between base and candidate.

Mechanism

Two facts combine:

1. coverage-report emits its cards array in nondeterministic order. The array order differs wholesale between runs (HashMap iteration).
2. coverage-parse-diff collects into a BTreeMap keyed by lowercased card_name, so last-occurrence-in-array wins. 30 of 35,657 rows share a lowercased card_name.

When a collision winner flips between the two runs being compared, the comparator sees two different oracle_text values for the same key, and its behavior on an oracle_text mismatch is to skip the card and increment oracle_changed.

Consequences:

• oracle_changed is not a reliable indicator of anything about the change under test.
• Skipped cards are silently excluded from cluster detection, so a genuine parse change on a colliding name could be dropped.
• The same ordering dependence perturbs coverage-data.json byte size via top_gaps / gap_bundles / parse_warning_patterns, so file-level digest comparison of coverage-data.json is also unreliable (observed: a 160-byte difference with identical parse content).

Evidence

Base vs candidate projections of the same pinned AtomicCards.json, built from two genuinely distinct engine builds (the six tool binaries differ by digest):

• card-data.json byte-identical: a25d3944efbeb96bcc78c2cc254d8857304e9a0340e23c921401e3b0b9547fb4, 99,502,331 bytes both sides. card-names.json likewise identical.
• Comparator: clusters: [], added_cards: [], removed_cards: [] — but oracle_changed: 1.
• Order-normalized (card_name, oracle_text, parse_details) multiset over all 35,657 rows is byte-identical on both sides (a0b5b54b0d6da232…), which covers the skipped card. So no parse output changed and the oracle_changed count was entirely an artifact.
• Non-degenerate input: 35,657 rows, 35,286 with non-empty parse_details, on both sides.

Suggested fixes

Any one of these would close it; the first two are independently worth doing:

1. Make coverage-report's cards array deterministically ordered (sort by a stable unique key). This also makes coverage-data.json digest-comparable, which is useful on its own.
2. Key the comparator on something unique rather than lowercased card_name — an oracle id, or (lowercased name, index) — so colliding names can't displace each other.
3. Report skipped cards explicitly instead of only counting them: emit which cards were skipped and why, so a nonzero count is actionable rather than ambiguous.

Note that js_downcase-style lowercasing concerns do not apply here — this is Rust-side to_lowercase() on both sides. The collision is genuine duplicate names, not an encoding mismatch.

Impact

Tooling/measurement only — no gameplay or rules impact. It matters because this comparator is the parse-diff evidence in the engine-implementer measurement flow, and a spurious oracle_changed either creates false alarm or, worse, silently hides a real cluster on a colliding name.