Testing

Testing

Use the smallest test that exercises a change, then validate its effects in the game. Unit tests, differential replays, renderer comparisons and live gameplay answer different questions; none substitutes for all the others.

Commands below run from the repository root. Build prerequisites and full runtime configuration are covered in Building. Captures and extracted game data are user-supplied local inputs.

Continuous integration

The CI workflow runs automatically on pushes to main and pull requests, including drafts, and can also be started manually. It uses only project source and authored fixtures, without a game dump.

Source checks cover distribution, Python tooling, oracle comparison, and VU code generation. For the public repository, builds run on Linux x86-64, Linux ARM64, and macOS ARM64. Independent runtime and graphics jobs compile the recompiler, runtime with FFmpeg, launcher stub, SDL GPU renderer and shaders, then run their respective suites. The VU build uses all committed authored .vublocks fixtures and requires the compiled region, loop, and branch paths to execute during differential checks. It also compiles the synthetic programs from ps2xTest/data/vu_programs.py as whole-program routines and requires each of them to run compiled and to match the interpreter (with NEON on the ARM runners, SSE2 on x86-64); the native IOP's SPU2 and DMA tests run in the same job.

The graphics jobs also build the launcher in tools/launcher and run its tests, and a Windows job does the same with MSVC. The Release workflow packages the launcher with everything the build needs but a compiler, for macOS (Apple silicon and Intel), Windows, and Linux (x86-64 and ARM64). Each push to main publishes a release named by the commit's short hash, with the source including submodules, FFmpeg's source and checksums. Pull requests that touch the launcher run all of it except the publishing, and the job summary shows the release they would make.

Linux graphics tests use Mesa's software Vulkan device; macOS uses Metal when available. Device-dependent skips appear in the logs and job summary. A skipped test does not validate rendering, and software Vulkan tests do not establish hardware-driver compatibility or playable field FPS. Full game builds, game-derived replays, and live visual checks remain local.

Standard GitHub-hosted runners are free for public repositories. While the repository is private, the full matrix is disabled and only the lightweight Linux launcher and CPU graphics checks run alongside tooling checks. Making it public triggers the expanded workflow automatically. Source checks have a 10-minute timeout, runtime builds have a 45-minute timeout, graphics builds have a 30-minute timeout, and newer runs cancel superseded runs for the same branch or PR. There are no schedules, artifact uploads, or build caches; private runs still count toward the owner's monthly allowance.

Source and tooling checks

python3 tools/check_source_distribution.py
python3 -m unittest discover -s tools/tests -v
python3 -m unittest discover -s tools/mvi -p 'test_*.py' -v

These checks use synthetic inputs and need no game dump. The distribution guard checks tracked filenames and common credential signatures in the current checkout; it does not inspect Git history or submodule contents, or determine licensing rights.

Graphics without a full game build

cmake -S tools/gsdump -B build/gsdump -G Ninja
cmake --build build/gsdump
ctest --test-dir build/gsdump --output-on-failure

This standalone project does not require the recompiled game corpus. SDL GPU tests require an available GPU device and can be skipped when one is unavailable. Check the test report: a skipped GPU test is not a successful hardware-renderer validation. The test configuration handles the supported offscreen setting; do not force it on macOS.

The gs-backend-regression target compares hardware and software output using local dumps. gs-replay-regression compares with local reference images. Configure DQ8_GSDUMP_DIR for your capture directory. These targets can skip when data is missing; confirm that the expected images were actually compared.

Runtime suites

In a configured full runtime build, here called build/native:

cmake --build build/native --target dq8-vu-tests dq8-vu-math-tests \
  dq8-memory-tests dq8-kernel-tests
ctest --test-dir build/native/tools/runtime \
  -R '^(vu|vu-math|memory|kernel)$' --output-on-failure

Additional targets cover guest pacing, render cadence, fibers, VU bounds and the native IOP's sound hardware (dq8-sound-tests). Build and run the relevant suite when changing those interfaces. Authored tests should prove that an optimized path is entered, not merely that its fallback still works.

The whole-program VU test needs its programs compiled into the runtime. Write them, configure with them and point the test at the same directory:

python3 thirdparty/PS2Recomp/ps2xTest/data/vu_programs.py "$PWD/build/vu-fixtures"
cmake -S . -B build/native "-DPS2X_VU_PROGRAM_PROFILES=$PWD/build/vu-fixtures"
cmake --build build/native --target dq8-vu-tests
PS2_VU_PROGRAM_FIXTURES="$PWD/build/vu-fixtures" PS2_VU_REQUIRE_PROGRAMS=1 \
  build/native/tools/runtime/dq8-vu-tests

It runs each program compiled and interpreted, whole and then stopped by budgets it outlasts and resumed in uneven slices. A game build's own recordings can go in the same list; only the directories named by PS2_VU_PROGRAM_FIXTURES are run by the test.

VU differential replay

Build dq8-vu-replay in the full runtime configuration. For a local capture prefix:

build/native/tools/runtime/dq8-vu-replay path/to/capture 1 --verify

The verifier compares state, VU data memory and emitted GIF packets against an independent raw-byte interpreter across resumed cycle budgets, and once more over a single uninterrupted run, which is the only one wide enough for whole-program routines to start. It also exercises code modification. A single normal replay's successful exit only establishes its own repeatability; it does not compare two binaries.

Preserve a baseline executable before changing code, then compare signatures and paired timing:

python3 tools/runtime/compare_vu_replays.py \
  path/to/baseline-replay path/to/candidate-replay path/to/captures \
  --repetitions 1000 --rounds 3

The comparator alternates binary order, checks signatures and reports per-capture medians. These replays do not establish original-hardware equivalence or full gameplay coverage. Add authored tests for delayed flags, raw tail execution, branch forwarding, exceptional arithmetic and every code-cache invalidation offset affected by a change.

A meaningful negative control deliberately removes the new safety rule and must produce the expected state or image mismatch. A test that only checks a coverage counter is useful, but insufficient as the sole negative control.

Trace comparison

tools/oracle/logdiff.py reports the first divergence between reference and candidate logs. Category filtering permits focused file/thread/audio comparisons, and normalization suppresses selected volatile values. Its self-test needs no game data:

python3 tools/oracle/logdiff.py --self-test

Ensure a reference actually contains the guest events being tested. An empty log, disabled patch or filtered-out category cannot prove equivalent execution. A fully automated end-to-end reference trace and RAM comparison pipeline is not currently established.

For renderer changes that should not alter results, record an ordered backend trace in the game (DQ8_GS_TRACE) and replay it with dq8-gs-trace-replay before and after the change. Besides the presented frames, the replay prints a hash of local memory at the end, which also catches differences that no presented frame shows.

Sound

PS2X_AUDIO_DUMP=path keeps a raw copy of everything played, as 48 kHz stereo signed 16-bit samples (-f s16le -ar 48000 -ac 2 for FFmpeg). The mix depends on timing, so compare it with a reference by listening and by spectrogram rather than byte for byte. Movie audio allows a stricter check: the chunks the game streams to the IOP should match the movie's audio track exactly.

The game feeds streamed speech a little every frame, so a run below full speed puts gaps in it. Judge speech only in runs that hold the game's frame rate.

Performance and visual acceptance

Preserve the source revision, binary identity, input set and configuration. Stop competing builds, replays and sampling during timing. Alternate baseline/candidate runs and separate profiler collection from sustained FPS windows.

Measure the same field scene and camera, with real movement and menu interaction checked separately. Count completed game renders as well as host presentations: presenting the same image repeatedly is not faster gameplay. Report scene, duration and counters, and distinguish VU-only or GS-only timing from whole-game FPS.

For graphics changes, compare actual images and inspect the live route. For timing changes, preserve guest clocks and input/audio ordering. Protect saves during repeated tests. Share reproduction instructions and authored fixtures publicly; keep copyrighted capture contents local.