Debugging Recipes

Debugging recipes

This page owns: short, copyable procedures for the questions that come up repeatedly.

Each recipe has been used in anger on this project. They assume the caveats on When your instrumentation lies — particularly that memory access is physical and unmunged.

---

1. Find the instruction that faulted

Question: something bugchecked with an address. Which instruction referenced it?

A boot takes thousands of ordinary page faults, so the trick is filtering.

python3 tools/mkbootscript.py --veneer <VENEER.EXE> --ckpt <pre-go.ckpt> \
        --bp '0x300:machine.cpu.dar == 0xEE315C98' \
        --bp 0x80679eac \
        --out tmp/boot.gs

The first is a conditional breakpoint on the data-storage vector, firing only for that address. Use the value as reported — DAR is munged, and so is the bugcheck's parameter.

The second is KeBugCheckEx as a backstop, so the run still stops usefully if the condition never matches. Get its address from the export table plus the load base:

python3 tools/pe-exports.py NTOSKRNL.EXE | grep KeBugCheckEx    # -> descriptor RVA

Each hit reports srr0 srr1 dar dsisr lr r1 r3..r6. SRR0 is your answer.

---

2. Identify which module a code address belongs to

Question: SRR0 is 0xEE326BC0. What is that?

If it is in a module the HAL listed, arithmetic is enough. Every boot prints the list:

HAL: module VIDEOPRT.SYS at 80705000 size 00007720

If it is not — a driver loaded after that point — fingerprint it:

# 1. translate and read the instructions (word munge!)
#    machine.cpu.mmu.translate(0xEE326BC0)  -> 0x004BEBC0
#    machine.memory.peek.l((0x4BEBC0 + 4*i) ^ 4)

# 2. take a branch-free run — position-independent
words = [0x63a40000, 0x63c50000, 0x3be10038, 0x63e60000, 0x38600004]
pat = b''.join(struct.pack('<I', w) for w in words)

# 3. search every driver on the disk
for f in glob.glob('drivers/*'):
    i = open(f, 'rb').read().find(pat)
    if i >= 0: print(f, hex(i))

Seven instructions identified MGA_MIL.SYS uniquely among 53. The file offset converts to an RVA, and runtime_address - RVA gives the module's load base — which unlocks everything else.

Extract the drivers first with tools/fatcat.py.

---

3. Watch one stack slot for a bad write

Question: a value in memory is wrong. Who wrote it?

--watch 0xE6E84324

Emits a write logpoint that reports the value, the writing PC and r1 on every hit.

Two caveats, both learned the hard way. Logpoint output goes to the log stream — for a daemon, its own stdout, not your client — so look in the daemon's log. And the address may need the munge applied depending on how the hook is wired; if nothing fires, try addr ^ 4.

When the watch does not work, the stack itself often answers it for free. Read the frame directly:

0xe6e84320  0xe6e84380    ← back chain
0xe6e84324  0xee31db04    ← 4(r1): the saved TOC slot, holding a code address

That single line was the whole of wall 49's proof.

---

4. See what the firmware handed the loader

Question: the loader is behaving as though it were told something odd. What was it told?

Turn on the veneer's argv tracing:

--vrdebug 0x2000

Argv[0]: multi(0)scsi(1)disk(0)rdisk(0)partition(1)\os\winnt40\osloader.exe
Argv[1]: OsLoader=multi(0)scsi(1)disk(0)rdisk(0)partition(1)\os\…
Argv[2]: SystemPartition=multi(0)scsi(1)disk(0)rdisk(0)partition(1)

One line answers "did the environment reach the loader?", which is otherwise several hours of inference. Add 0x1000 to trace every read and seek — that is how "is it reading the partition or the whole disk?" gets settled. → The veneer §7

---

5. Check a disk image from the host

Question: is the file on the disk the one I think it is?

python3 tools/fatls.py  tmp/nt-installed.img                     # what is there
python3 tools/fatcat.py tmp/nt-installed.img 1 '\OS\WINNT40\HAL.DLL' /tmp/x.dll
cmp build/hal.dll /tmp/x.dll && echo same

And the one that has saved two debugging sessions — check the image is not truncated:

python3 tools/savedisk.py tmp/ckpt-daemon tmp/out.img --check

→ Capturing the installed image

---

6. Prove a build has no misaligned accesses

Question: did the compiler fold my careful byte-at-a-time code back into a wide access?

llvm-objdump -d build/hal.elf | grep -E 'lwbrx|stwbrx|lhbrx|sthbrx'

Nothing printed is the pass condition. This catches the byte-reversed load shape, which the Makefile's store-merging flag does not cover. → Little-endian PowerPC §5

---

7. Ask Open Firmware directly

Question: is the firmware doing what I think, or is something above it at fault?

Drop to the 0 > prompt and try it by hand. This settled the :N raw-open question in one run, after a day of inference:

variable ih
" /bandit@F2000000/53c825@12/sd@0,0:1" open-dev ih !
600000 200 " read" ih @ $call-method .
600000 10 dump
ih @ close-dev

Two practical notes. The serial input FIFO holds about sixteen characters, so long lines lose their head — feed them in chunks. And dump on memory the firmware has not claimed throws DEFAULT CATCH!; use alloc-mem for a buffer. → Open Firmware §6

---

8. Recompute an address that moved

Question: a breakpoint or poke that worked last week now does nothing.

Almost always this: changing the HAL's size moves every module loaded after it. Any hard-coded runtime address in another module is invalidated by an unrelated HAL rebuild.

Recompute from the module list the HAL prints on every boot:

HAL: module VIDEOPRT.SYS at 80705000 size 00007720
                            ^^^^^^^^ this moved

base + RVA for a breakpoint; (physical + RVA) ^ 4 for a word poke. → Iterating on the HAL §7

---

9. Use the daemon after the run

Question: the run is over and I wish I had printed one more thing.

You may not need to re-run. The daemon holds the machine after a script finishes — registers, memory and screen are all still there:

python3 tools/gsh.py 'echo "pc=${machine.cpu.pc} r2=${machine.cpu.r2}"'

Execution is deterministic, so a re-run reproduces the same state exactly — but inspecting what is already loaded is free, and more than once the answer was sitting in a daemon nobody had touched.

---

Next

• The emulator shell — the primitives these are built from.
• Decoding a bugcheck — recipes 1 and 2 in context.