Testing Strategy

Testing Strategy

Test ROMs are the spec. When the docs and a passing test ROM disagree, the ROM wins — the doc gets updated, not the code. For accuracy work, the failing ROM's expectation is pinned first, then implementation continues until it passes. Stella is the behavioral oracle: when prose docs and Stella disagree about a hardware quirk, Stella plus a confirming test ROM wins.

The layers

• Layer 1 — unit (per crate, >90% line coverage on the chip crates). Register decode, board hotspots, timer prescale, polynomial-counter state.
• Layer 2 — CPU golden-log. The Klaus Dormann functional and decimal-mode (BCD) test suites, plus the SingleStepTests/ProcessorTests 6502 set (per-opcode, per-cycle bus-activity vectors, including undocumented NMOS opcodes). A golden-log differ captures a real (PC, A, X, Y, SP, P, cycle) record per retired instruction and diffs it against a loaded reference log — including a genuinely independent one: 20,000 instructions of the Klaus functional test captured from Gopher2600's own separate CPU implementation, diffed register-for-register and cycle-for-cycle. Two independently-implemented 6502 cores agreeing this closely is real external validation, distinct from a test ROM's own internal self-check.
• Layer 3 — test-ROM corpus. TIA timing/draw ROMs (beam-racing, RESPx positioning, HMOVE-comb, collisions) verified against Stella-derived reference scanline buffers, plus the Stella regression corpus for mapper/audio/per-quirk behavior. A dedicated full-System conformance-suite runner exists but stays permanently a stub for TIA/RIOT-specific ROMs — researched twice, no freely-redistributable 2600-specific TIA/RIOT test-ROM corpus exists anywhere (Gopher2600's own README makes the same admission), and the one official corpus that does exist (the Atari Diagnostic Test Cartridge) is service-center software, not freely redistributable. This is a real, permanent scope boundary. TIA/RIOT accuracy work instead continues via a differential-oracle method against specific known-hard commercial titles — the same technique that found and fixed the real Frogger WSYNC jitter bug and the Pitfall II RIOT-timer boot-loop bug described on Lockstep-Scheduler and RIOT.
• Layer 4 — accuracy battery. Runs the bundled golden-log oracles and records each into a real pass/fail score, gated in CI so a regression in either can never silently ship. Currently 2/2, 100%.
• Layer 5 — visual golden + screenshots. Diffs the composed scanline buffer against a committed golden image, with real tolerance-aware comparison alongside an exact byte-diff. Only CC0/public-domain ROMs and screenshots are committed to the repository; commercial ROM dumps used for local cross-checking live in a gitignored directory and are never committed.

The honesty gate

A dedicated test enforces the headline-claim invariant across the whole cart catalogue: every board the accuracy battery covers must report a Core or Curated tier — never BestEffort. A BestEffort board may carry reference screenshots, but it can structurally never inflate the accuracy number. See Cart-Catalogue and Architecture-Decision-Records ADR 0003.

Determinism in tests

Every harness run seeds the System explicitly; same seed + ROM + input sequence yields a bit-identical scanline buffer and audio (see Architecture-Decision-Records ADR 0004). This is what makes the golden-log differ and the snapshot comparator stable across repeated runs.

Differential-oracle cross-checking

Beyond the committed test-ROM corpus, Rusty2600's accuracy work leans on studying two open-source references (never vendored — both are GPL-licensed, incompatible with this project's dual MIT/Apache-2.0 license, so they're read for behavioral cross-checking only, never copied): Stella, the canonical C++ Atari 2600 emulator, and Gopher2600, a Go implementation used as a headless differential-testing harness — building it and running it directly against the same ROM as Rusty2600, then comparing instruction-by-instruction traces, is what has found the project's hardest real bugs.

See also

Architecture-Overview · Cart-Catalogue · Architecture-Decision-Records · Reference-Materials