Firmware Testing

Important

Two of the four directories on this page are in a release; two are not. test/host/ and test/qc/ are in every release up to and including v1.2.0, the newest. test/live/, test/bench/, DEBUGGING.md, the Doxyfile and the PICO9918_LIVE_TEST build option exist only on the development branch: git ls-tree main test/ returns CMakeLists.txt, host and qc, and nothing else. If you have cloned a release tag rather than the branch, the live harness and the benchmark ROM are not there to run.

The firmware carries its own test infrastructure: four directories under test/, each answering a different question, plus a Doxygen target for the API documentation. This page is a tour of what is there and how to run it, and assumes you can already build the firmware. See Building from Source. DEBUGGING.md in the repository root is the front door: it maps each question to the tool that answers it, says what each one costs, and lists the traps that have already cost this project time. Start there if you do not yet know which tool you want.

What's in the test tree

Directory	Question it answers	What you need to run it
test/live/	Did the renderer draw the right pixels, and did every scanline fit?	A Pico debug probe, a PICO9918_LIVE_TEST=ON firmware build, Python 3
test/bench/	What does a scanline cost on a shipping build driven by a real host?	A TI-99/4A, ColecoVision or MSX, a cartridge, and the performance overlay
test/host/	Does the board talk to a host at all?	A second Pico wired to the board's TMS9918A pins, and a display
test/qc/	Was this board assembled correctly?	Three jumpers and a USB serial terminal

Only test/host/ and test/qc/ are part of the ordinary firmware build: test/CMakeLists.txt adds both, and the root CMakeLists.txt adds test/ in the plain firmware configuration. test/bench/ is a separate CMake project, because it builds Z80 and TMS9900 cartridge ROMs rather than anything for the RP2040. test/live/ builds nothing at all: it is Python driving a live-test firmware build over SWD, and it is the only one of the four that runs without a person watching a screen. It also answers both halves of its question in a single pass, deliberately: an over-budget scanline is skipped rather than drawn late, so a scene that does not fit comes back as pixels that are wrong, and correctness and headroom cannot be separated on this hardware.

The live SWD harness

A Pico debug probe drives the board over SWD: probe SWCLK, SWDIO and GND to the board's SWCLK, SWDIO and GND. Scene setup writes VDP registers and VRAM straight into tms9918Inst, so there is no cartridge, no host machine and no ROM build; readback is the exact index buffer the renderer produced; flashing is openocd's job too. A cycle is edit, flash, capture, assert. The debug connector carries no power, so the board needs its own: the probe's test points will do. openocd comes from the Pico SDK install and needs no separate download, and the target config follows the silicon (rp2040.cfg or rp2350.cfg), not the UF2 that was built. The reference bench rig is a ZIF socket on a carrier board rather than a console, with CD0-7, /CSR, /CSW, MODE and RST each pulled up 4.7K to +5V: both chip selects idle high, so no bus cycle ever begins. test/live/README.md explains why each pull-up is there.

Building a live-test firmware

cmake -S . -B build-live -G Ninja -DPICO9918_BUILD_COMBINED=ON -DPICO9918_LIVE_TEST=ON \
      -DPICO9918_VERSION_SUFFIX=live
cmake --build build-live --target combined

python test/live/live9918.py flash build-live/pico9918pro/src/pico9918pro-v1-3-0-live.elf

PICO9918_LIVE_TEST (see Firmware Configuration Options, where it is marked as development-branch only) adds 288 bytes of .text: one call per scanline which, while a capture is armed, hands the line to a DMA that copies it into a 48-row buffer. 48 divides every frame height the renderer produces (192, 240, 384 and 480) so a frame is read in whole passes. That copy also runs through the DMA sniffer, which CRC-32s it in hardware, so a reader comparing against a reference it already holds asks for the frame CRC and is done without transferring a pixel. Never take timing readings from such a build: the copy costs about a microsecond a line while armed, and the buffer inflates every SRAM figure.

Running the suite

runner.py is the whole rig in one command, in one openocd session, in the order that works:

python runner.py --board 2040 --quick         # canaries and properties, after every edit
python runner.py --board 2040 --save --report # every stage; writes runs/ and reports/
python runner.py --board 2040 --clock 1       # reboot at 302 MHz and measure there

The stages are d4, freeze, diag, perf, perf-panels, scroll, colour, ecm and t80-8bpp, selectable with --only and --skip; --quick runs d4, freeze, scroll, colour and ecm. --board takes pro or 2040 and picks an ELF, not a board: every session checks the chip on the probe before it writes and refuses a mismatch, because program otherwise reports success and the target then answers nothing at all. --clock is the one knob that reboots the board: the presets are the firmware's own, 0 = 252 MHz, 1 = 302 MHz, 2 = 352 MHz, and clocks.c reads the value once out of flash before the renderer starts, so it cannot be changed in the running copy (see Clock Presets).

A run leaves runs/<board>-<commit>.json: every scene's state, the rows it dropped, what it cost and what each property asserted, stamped with the commit, branch, dirty flag, board, firmware version, the clock read off the board, the toolchain, every PICO9918_* option the ELF was configured with, and its section sizes. Records are committed, so a comparison can reach back through the branch rather than only to whatever is on this disk.

Goldens

python freeze.py                 # compare every scene against test/live/golden/
python freeze.py --update        # write the references
python freeze.py --canaries      # the scenes that drop a line before any average moves
python freeze.py --audit         # which features are still missing (needs no board)
python freeze.py --diag          # with every overlay panel on: what the overlay costs, per line

A positional filter selects scenes whose name contains it, and --png DIR writes a PNG per scene when you want to look rather than assert. A reference is keyed on the width of the line that produced it, so there are two directories (golden/ for the 256-byte line and golden-w512/ for the 8bpp 80-column tier) and a build only ever writes its own width's. A golden is not automatically the right answer, so each scene declares itself. Frozen means any change is a regression. Provisional means the scene exercises something the F18A does and this firmware does not, so its capture records an absence; --audit checks each one still matches the scene it should be identical to while the feature is missing. Over budget on a named board means it renders correctly and does not fit, which is reported rather than failed.

Important

freeze.py --update overwrites the committed references in place. A provisional scene's golden must change when its feature lands, but re-freezing is how that change gets reviewed rather than absorbed, so run --update deliberately, one feature at a time, and read the diff.

Properties

A golden records whatever the firmware did, which is the wrong instrument for a feature that has just arrived: it freezes the implementation's own opinion. So each new feature also gets a property taken from the hardware reference and asserted over a sweep, with the goldens there to stop it regressing afterwards. Each property blanks all 64 registers and all of VRAM before writing its own, because the state it inherits comes from the previous run, stage order cannot fix order-dependence.

python properties/test_d4.py           # layer arbitration, all 192 rows against an exact image
python properties/test_text_scroll.py  # a scrolled row is the unscrolled one, rotated
python properties/test_text_colour.py  # every pixel takes the colour its layer's sub-palette gives
python properties/test_text_ecm.py     # a text cell above ECM0 is a six-pixel-wide ECM tile

Scenes, reports and timing

scenes.py holds the hand-written scenes (every mode crossed with ECM, the layer enables, both scrolls, the page wraps, sprites, priority and the bitmap layer) and every VDP dump in test/live/dumps/ becomes one more, with no code and no registration. What a scene exercises is derived from its register file rather than declared beside it, which is why it also works for the dumps.

python scenes.py                              # the feature matrix, no board needed
python web/console.py --board 2040            # opens http://127.0.0.1:8919/
python web/report.py --compare 2040-aba934b 2040-bce997d
python perf.py --save before                  # RENDER and FRAME, on the control build
python perf.py --against before               # ...and on the changed one

web/console.py is a browser front end for the board: a grid of every scene, click one to put it on the screen, drag a VDP dump onto the page to see it live, plus register controls for scroll, layers, ECM, palette and the diagnostic panels. It holds one openocd session for its lifetime, so nothing else can talk to the board while it runs. web/report.html is the report: open it and drop one record on it, or two to compare; it fetches nothing and needs no server. perf.py reads RENDER and FRAME, the same two numbers the Performance Overlay shows, over SWD with no overlay to photograph; both builds in a comparison must be live-test builds or it is not a control.

The benchmark ROM

test/bench/ builds three cartridge ROMs, one per host, from one source, each with eleven fixed scenes, advanced with any key or the fire button. It exists for the timing the live harness cannot take: a live-test build's absolute numbers are inflated by the capture copy, so they are a control against another live-test build rather than a figure for a shipping one. Every scene here is static once set up, so a reading off the diagnostic overlay is repeatable.

The CVBasic toolchain comes from the configurator build, so build that once (any target under configurator_all) and the tools land in build/pico9918/external. Then:

cmake -S test/bench -B build-bench -G Ninja
cmake --build build-bench

ROMs appear in build-bench/dist: pico9918bench_ti99_8.bin (TI-99/4A via FinalGROM 99, and it needs the 32K expansion), pico9918bench_cv.rom (ColecoVision) and pico9918bench_msx.rom (MSX). Pass -DPICO9918_TOOLS_DIR=<other build>/external to borrow the tools from a different firmware build tree. The backdrop colour identifies the scene (scene N runs with backdrop N + 1) and test/bench/README.md lists what each is for and, more usefully, what each should look like: the shapes are chosen so that "correct" is a local check rather than a measurement, and in scenes 8 to 10 any blue on screen at all is a defect.

The host bus test-bed

pico9918test is a second Pico standing in for the host CPU. It generates GROMCLK and CPUCLK, drives the TMS9918A bus (/CSR, /CSW, MODE, CD0-7), sets up a Graphics II screen with sprites, and animates them from the vertical interrupt. It is the only thing in test/ that goes through the host interface; the live harness bypasses the bus entirely and the benchmark ROM assumes a working one.

cmake --build build --target pico9918test

Flash pico9918test.uf2 to the Pico that will play the host, wire it to the PICO9918's TMS9918A pins per the GPIO_* defines in test.c (which are that Pico's own numbers, not the firmware's) and give the PICO9918 its own power and a display. Expect the BREAKOUT image in Graphics II with Hello, World! spelled out in sprites twice over, drifting horizontally on a sine and downward one pixel every two frames.

It is not an assertion: nothing here compares anything, the test is a person looking at a screen, and it cannot distinguish a wrong picture from a frozen one. It does not test bus timing either (the delays in writeTo9918() are sleep_us(0)) so it exercises the interface at whatever speed the compiler and the GPIO block produce, not at the datasheet's margins. For the specified numbers see Host Bus Timing.

Warning

pico9918test and pico9918qc are standalone UF2s that replace the firmware on whatever RP2040 they are flashed to, and neither produces video. pico9918test belongs on the second Pico, never on the PICO9918. Flash the real firmware back when you are done with pico9918qc.

Board QC loopback

pico9918qc checks that an assembled board is wired the way the schematic says. It is a loopback test: three jumpers tie an output to an input, and the tool drives one end and reads the other. Each pair puts an output on one side of the board's buffers and an input on the other, so a successful read proves the pin, the buffer and the trace between them:

Drive	Read back
CPUCLK	/CSR
GROMCLK	/CSW
/INT	MODE

cmake --build build --target pico9918qc

Flash pico9918qc.uf2 and open the USB serial port: pico_enable_stdio_usb is on for this target and the output is the whole result. It runs forever, restarting from the top after each pass. Six loopback checks come first, two per pair (drive high, expect high; drive low, expect low), each printing OK or FAILED!; if any fails the tool prints Initial check failed. Halting. and stops. Then a walking-1 on CD0-7 runs twice, once with /CSR asserted and once released, so the two passes differ in the direction U5 is driving, which catches a transceiver whose direction control is dead or inverted, a fault the first pass alone reports as a clean board. That second pass needs a meter, a scope or a logic analyser on the host-side pins; the tool prints what it is driving and cannot see the result itself.

PCB_MAJOR_VERSION and PCB_MINOR_VERSION at the top of qc.c select the pin map, and GROMCLK and CPUCLK moved at PCB v0.4, so check the constants match the board in front of you before believing a failure: a wrong pin map looks exactly like a fault. This tool answers the one question no software test can, which makes it the first thing to run on a board that behaves strangely, before spending any time on the firmware or on the software-side checks in Troubleshooting.

API documentation

Doxygen output is a separate target, built from the firmware source plus docs/mainpage.md, BUILDING.md and DEBUGGING.md:

cmake --build build --target docs

HTML lands in build/doxygen/html whichever build directory invoked the target, because Doxygen resolves its paths against the source root. The target only exists when Doxygen is installed and is never part of all, so a build without it configures and builds normally. Doxyfile lists only the settings that differ from the defaults; statics and private members are extracted deliberately, since the renderer's mode emitters are static and are the part a reader most needs explained.

Debugging a running board

The shipping firmware opens no stdio at all (src/CMakeLists.txt disables both UART and USB) so there is no printf channel to route. Debugging is SWD, the diagnostic overlays, or both:

• SWD, via the live harness. live9918.py flash <elf> programs the board, shot frame.png --elf <elf> captures a frame to PNG and regs --elf <elf> dumps the register file; the Live class is usable directly from a script too. It refuses to read a board running an image other than the ELF you named: everything here is addressed by symbol, so a stale image does not fail, it reads the wrong addresses.
• The diagnostic overlays, which are stored configuration and so survive a flash. That matters when measuring: scenes.apply turns all five flags off for every scene and perf.py --panels turns them back on deliberately. See Diagnostic Overlays.
• DEBUGGING.md for the rest: how to read RENDER against FRAME, why a dropped row is the sharpest headroom test there is, the rules for measuring Cortex-M0+ code, a table of traps that have already cost time, and an explicit list of what nothing here covers: the host bus, everything past the index buffer, and sprites at 8bpp.

Caution

Do not let the firmware save its own configuration with a debugger attached. CONF_SAVE_FORCED makes the GPU core erase and program flash while core 0 renders out of it; the board then stops after one frame and answers zeros for every flash address until the sector is rewritten from outside. The firmware takes that route on its own account whenever the stored configuration is blank or older than the build, so a board can arrive in this state on the first boot after a version bump. Any runner.py --clock N run repairs it, because the harness writes the sector itself with both cores halted. Erasing the sector alone does not, the next boot only retries the same save.

If a board stops answering the probe entirely, the RP2040's rescue debug port resets its power state machine so the bootrom halts and takes fresh code:

openocd -s <sdk>/openocd/0.12.0+dev/scripts -f interface/cmsis-dap.cfg \
        -c "adapter speed 1000" -c "set RESCUE 1" -f target/rp2040.cfg -c "exit"
python live9918.py flash ../../build-live/pico9918/src/pico9918-v1-3-0-live.elf

Related

• Building from Source: prerequisites, build options and outputs, including PICO9918_LIVE_TEST
• Firmware Architecture: cores, PIO, DMA and the render path the harness measures
• Diagnostic Overlays: the RENDER and FRAME counters perf.py reads over SWD
• Host Bus Timing: the specified read and write cycles the bus test-bed does not check
• Troubleshooting: symptom-first checks for a board that misbehaves in a host
• test/: the test tree on the default branch, which holds host/ and qc/ only