Reverse Engineering with Lockstep

Reverse-Engineering with Lockstep

The single most valuable technique in building this core was lockstep tracing against a reference emulator. If you're writing a CPU emulator for an under-documented machine, this is the tool that turns "it's stuck somewhere in 20,000 instructions" into "instruction N, this register is wrong."

The idea

Run a known-good emulator and your own emulator over the exact same ROM, dump a per-instruction trace from each, and diff them. The first instruction where the program counters disagree is — almost always — the exact instruction your CPU got wrong.

For the Game.com the reference was MAME's gamecom driver (whose SM8500 CPU code is BSD-3 and which Tigerbyte's implementation credits). MAME can dump a CPU trace from its debugger:

mame gamecom -debug -debugscript trace.do ...
# trace.do:
trace boot.tr,maincpu,noloop
go

Two non-obvious things matter:

• Use noloop. By default the trace collapses repeated loops into a (loops for N instructions) marker. That desynchronizes a naive line-by-line diff and produces a fake divergence. (This genuinely cost an hour before the penny dropped.)
• Diff only the program counter (the leading XXXX: of each line), since the two emulators' disassembly text differs cosmetically. Matching PCs means matching control flow; the first mismatch is the bug.

What it found, in minutes

After wiring up the lockstep, the traces matched for thousands of instructions and then diverged at one instruction: a word-ALU opcode group was decoding with the wrong operand length (one byte instead of two), because it had been given a different addressing form than its 8-bit sibling. Two-line fix. That single bug had been silently corrupting a table the kernel builds, which is why the boot "hung" far downstream with no obvious cause — exactly the kind of thing that's near-impossible to find by inspection but trivial to find by lockstep.

The same diff also cleared a suspected bug: a "stuck" cartridge boot turned out to match the reference exactly — the reference doesn't auto-launch a cartridge either, so the behavior was correct, not broken. Lockstep tells you what's wrong and confirms what's right.

Caveats

• Lockstep is exact only for deterministic execution. Once interrupts start firing, the two emulators' timing diverges (different cycle counts → interrupts land on different instructions), so the clean diff window is the boot/init phase before the first interrupt. That's still tens of thousands of instructions — more than enough to shake out CPU bugs.
• It validates the CPU, not the peripherals; for those you still read the reference's source and reason about the hardware.