Testing Strategy

Testing Strategy

Reference: docs/testing-strategy.md.

The governing rule: test ROMs are the spec. When a doc and a passing test ROM disagree, the ROM wins and the doc gets updated. For accuracy work, the failing ROM expectation is pinned first, then implementation proceeds until it passes.

Why a composed oracle

RustyNES (this project's NES sibling) has a single canonical 139-test AccuracyCoin battery. The SNES has no equivalent single battery, and no Nintendulator-style textual golden CPU log exists for the 65816 — so RustySNES's oracle is composed of two layers instead of one battery.

Layer 1 — the per-opcode oracle (both CPUs)

SingleStepTests 65816 + spc700 JSON: per-opcode, every addressing mode, 8-bit and 16-bit, native and emulation mode, with cycle-by-cycle bus-pin traces. The 65816 set ships with no license, so RustySNES self-generates its own committed oracle of record (cross-validated against the unlicensed upstream set, gilyon, and ares before freeze — Architecture-Decision-Records, ADR 0005) rather than depending on the unlicensed set directly.

Layer 2 — committable on-cart system tests

• gilyon/snes-tests (MIT) — pass/fail .sfc ROMs for both CPUs with golden result tables. Committed.
• undisbeliever/snes-test-roms (Zlib) — PPU/DMA/HDMA hardware behavior, including documented hardware glitches. Committed.
• blargg spc_* — cycle-accurate SPC700/S-DSP tests. License unstated; kept external/reference tier only.

Layer 3 — the accuracy battery

The composed pass-rate gate across the layers above. Target: ≥90% by v1.0.0, 100% the goal. Hard residuals that need the fractional-timebase refactor (Architecture-Decision-Records, ADR 0002) are explicitly documented as deferred, not silently point-fixed.

Layer 4 — visual golden + screenshots

Committed framebuffer/audio hashes in tests/golden/, plus per-coprocessor commercial-ROM regression harnesses (e.g. superfx_oncart, sa1_oncart, dsp1_oncart) that boot a real cartridge on the full System and assert detection, liveness, and a deterministic golden hash. Commercial ROM images themselves are never committed — only derived screenshots, hashes, and .snap files, kept in the gitignored tests/roms/external/ tree.

Layer 5 — determinism

A save-state round-trip (save → run N frames on a fork → load → run the same N frames on the original) must be bit-identical in both framebuffer and audio (Architecture-Decision-Records, ADR 0004). This is the same contract rewind, run-ahead, and (eventually) rollback netplay all depend on.

The honesty gate

Once boards are tiered (see Cartridge-and-Coprocessors), a dedicated test fails if any BestEffort board backs the accuracy oracle. BestEffort boards may carry reference screenshots for regression purposes, but they never inflate the headline accuracy number.

Never

• Never commit commercial Nintendo ROMs — only derived, non-copyrightable artifacts (hashes, screenshots).
• Never vendor LGPL/GPL/unlicensed test corpora into the MIT/Apache-2.0 tree; keep them external/reference-only.

Running the suites locally

cargo test --workspace                      # unit + integration tests
cargo test --workspace --features test-roms # adds the committed test-ROM oracle

See Building-from-Source for the full local quality gate, and Roadmap for where the accuracy battery push (v0.5.0 "Fidelity") sits in the release ladder.