Debugging

Debugging

Techniques for diagnosing NyxOS, in rough order of how often they are useful: the serial log, QEMU's interrupt trace, the kernel shell, the self-test suite, and the panic screen.

See also: Building, Troubleshooting, Boot-Process, Kernel-Data-Structures

The serial log

The first place to look. In gui, net and debug modes every kernel printf goes to qemu_serial.txt in the repository root; in serial mode it goes to stdio.

CODE — Watch the log while the VM runs

host $ tail -f qemu_serial.txt

Reading a boot log

A healthy boot is a fixed sequence. Knowing what it looks like makes a missing line obvious.

CODE — Boot transcript (captured on a v5.9.48 build, QEMU with -smp 4 -nic user,model=rtl8139 -hda ext2-test.img; the message set is current)

[INIT] Global Descriptor Table...
[INIT] Interrupt Descriptor Table...
[INIT] Interrupt Service Routines...
[INIT] Interrupt Requests...
[INIT] Serial Port...
[INIT] SSE/FPU for userspace...
[INIT] Memory detected: 511 MB
[INIT] Physical Memory Manager...
[MEM] mmap 7 regions: 125986 pages free (492 MB usable), RAM top 511 MB
[INIT] Paging...
[PAGING] Allocating PML4 table...
[PAGING] Identity-mapping 512 MB (RAM 511 MB)...
[PAGING] Loading CR3 with 0x13be000
[PAGING] NXE enabled.
[PAGING] CR0.WP enabled.
[PAGING] Enabled successfully.
[INIT] CPU protections (SMEP/SMAP)...
[CPU] SMEP=unavailable SMAP=unavailable
[…nyxfetch banner…]
[INIT] APIC...
[APIC] Version: 0x14, Max LVT: 5
[APIC] BSP APIC ID: 0
[APIC] I/O APIC ID: 0, Version: 0x20, Max IRQs: 24
[APIC] Legacy PIC disabled, IRQs routed through I/O APIC.
[INIT] Kernel Heap...
[INIT] Slab Allocator...
[INIT] SMP...
[SMP] 3 AP(s) found via CPUID
[SMP] AP 1 (APIC 1) started
[SMP] AP 2 (APIC 2) started
[SMP] AP 3 (APIC 3) started
[SMP] 4 CPU(s) online
[INIT] VBE (Bochs)...
[VBE] Setting mode: 1024x768x32
[VBE] LFB physical from PCI BAR0: 0xfd000000
[VBE] LFB mapped at e0000000 (768 pages)
[INIT] IST stacks: 4 CPU(s) x 2 (DF+NMI); CPU0 DF=0xffffff800028eb60 CPU3 DF=0xffffff800029ab60
[TIMER] 1000 Hz (interrupt-driven)
[INIT] Process Manager...
[SMP] 3 AP worker thread(s) pinned
[INIT] System Calls...
[INIT] Virtual File System...
[MODULES] 0 module(s) at 0x1
[INIT] EXT2 Filesystem...
[NET] Loopback: 127.0.0.1
[RTL8139] Found at 0:3.0
[RTL8139] MAC: 52:54:00:12:34:56
[RTL8139] Initialized successfully
[NET] Stack ready
[INIT] Auto-DHCP on eth0...
[DHCP] DISCOVER sent (xid=0x12345679)
[DHCP] OFFER: IP 10.0.2.15 from server 10.0.2.2
[DHCP] REQUEST sent
[DHCP] ACK: IP=10.0.2.15 mask=255.255.255.0 gw=10.0.2.2
[MOUSE] Initialized (wheel)
[SB16] DSP reset failed
[INIT] SB16 not detected (QEMU -soundhw sb16 required)
[INIT] Enabling interrupts (sti)...

[READY] NyxOS initialized successfully.

[INITRAMFS] Data at 0x140980, 1146468 bytes
[INITRAMFS] Loaded 64 files
[LIBC] shared libc mapped at 0x30000000
[INIT] Registered init PID=6 (scheduler will start it)
[EXT2] Found: 16384 blocks, 4096 inodes, block size 1024
[EXT2] Mounted /mnt (writable, persistent) - 11733 KB free.
[AUTH] Persistent user file found at /etc/passwd
[LOGIN] Starting login screen...
[LOGIN] OK.
[COMP] calling compositor_run
[FB] double buffering on — back buffer 1024x768 (3072 KB)

What the last line tells you

The boot dies where the log stops. Match the final line against the stage it belongs to:

Last line	Reached	Suspect
[INIT] Paging...	Page tables being built	Bad PML4 setup, memory map misparsed
[PAGING] Loading CR3	About to enable paging	The new tables do not map the code that is running
[INIT] SMP...	AP bringup	Trampoline, or an AP faulting silently
[VBE] Setting mode	Display	Unsupported mode; the kernel should fall back to text
[TIMER] 1000 Hz	Timer live	If ticks never advance, check the I/O APIC pin
[INIT] Enabling interrupts (sti)	First interrupt about to land	A broken handler faults immediately here
[INITRAMFS] Loading...	Unpacking the root filesystem	Corrupt or stale initramfs_data.h
[LOGIN] Starting login screen	Framebuffer login	Draw or input path
[COMP] calling compositor_run	Desktop	Compositor

Note

[CPU] SMEP=unavailable SMAP=unavailable and [SB16] DSP reset failed are normal under a default QEMU qemu64 CPU without an SB16 device. They are not errors.

QEMU interrupt tracing

CODE — Boot with an interrupt and reset trace

host $ ./run.ps1 -Mode debug

This adds -d cpu_reset,int, which logs every interrupt and every CPU reset with the full register state.

Important

Use this before adding printf calls to a suspected race. Serial output changes timing enough to move or hide the bug; the interrupt trace does not. Several bugs in this codebase — the irq_common register clobber, the iretq SS selector, the null-stack scheduler switch — were only findable this way.

Reading a triple fault

A triple fault appears as a chain, and the chain tells you the story:

     0: v=0e e=0002 i=0 cpl=3 IP=0033:0000000000010049 ...   #PF in ring 3
     1: v=08 e=0000 i=0 cpl=0 IP=0010:ffffff8000012a40 ...   #DF — the #PF handler faulted
check_exception old: 0x8 new 0xe                              → triple fault, CPU resets

Vector	Name	Usual cause in NyxOS
0x06	#UD	Executing garbage — a corrupted register used as a jump target; syscall without EFER.SCE
0x08	#DF	The fault handler itself faulted — usually a bad IST stack
0x0D	#GP	Non-canonical address, bad segment selector, iretq popping a code selector into SS
0x0E	#PF	Unmapped page. e= decodes the reason; cpl= tells you the ring

The e= error code bits for #PF: bit 0 present, bit 1 write, bit 2 user, bit 3 reserved-bit set, bit 4 instruction fetch.

The panic screen

An unrecoverable kernel fault draws a graphical stop screen with the faulting RIP, CS, ring and error code, rather than freezing silently.

CODE — Trigger a panic deliberately

nyx> crash

Use crash to confirm the panic path still works after touching the IDT, the TSS or the IST stacks.

Note

A ring-3 fault does not panic. It becomes SIGSEGV/SIGFPE/SIGILL and is delivered to the process. If a user program faults and the machine panics, that is itself the bug. See Security.

Locating a kernel address

The panic screen and the interrupt trace both give you a RIP. To turn it into a function:

CODE — Resolve an address to a symbol

host $ nm -n kernel/nyx-kernel.bin | less
host $ objdump -d kernel/nyx-kernel.bin | less

Kernel code is linked low and executed through the higher-half alias, so subtract KERNEL_BASE (0xFFFFFF8000000000) from a higher-half RIP before looking it up.

The kernel shell

CODE — Interactive kernel shell over stdio

host $ ./run.ps1 -Mode serial

Command	Shows
mem	Physical page usage and heap statistics
ps	Process table — pid, ppid, state, CPU time, command
cpus	Per-core state and tick counters
jobs	Background jobs
df	Mounted filesystem usage
mount	Mount table
ifconfig	Interfaces, addresses, link state
hexdump <addr> [bytes]	Raw memory
tree [path]	Filesystem tree
version, uname, nyxfetch	Build identification

The self-test suite

Every self-test is a known-answer test against published vectors, and returns non-zero on failure. Run them after touching the relevant subsystem — they are much faster than reproducing a failure at the protocol level.

Area	Commands
Memory and paging	cowtest
SMP	smpstress [secs], smpthreads [secs], smpuser [n], tlbtest
Scheduler	mtdemo
Networking	tcploop, tcpdrop <n>, tcpserve
Cryptography	x25519test prftest tlskeytest gcmtest tlsrectest csprngtest dertest p256test p384test skp384test rsatest sha512test chaintest
Images	deflatetest pngtest bmptest giftest jpegtest
Browser	formtest

tlbtest is worth singling out: it proves cross-CPU shootdown the only way that means anything, by first showing CPU 1 reading a stale value after a remap with no IPI, then the fresh one.

Debugging userspace

Technique	How
Exit status	The shell prints [exec] PID n exited (code c)
Signal death	Status 128 + signo — 139 is SIGSEGV, 143 is SIGTERM
Mappings	pmap <pid> reads /proc/<pid>/maps
Memory	free reads /proc/meminfo
Process state	cat /proc/<pid>/status
Live view	top
Fd leaks	The kernel logs [reap] force-closed leaked fd(s)
Syscall tracing	SYS_EXIT already logs [USER] exit(n); add a printf to the relevant case in syscall.c

The dedicated reproducers in user/ are often the quickest route: pstorm and bigread for pipelines, netstorm for concurrent sockets, vfsfill for VFS capacity, stacktest for stack growth, vmtest for the address boundary.

Reproducing intermittent bugs

The hard bugs in this project have all been timing-dependent. What worked:

1. Find a load that makes it frequent. The *storm programs exist for this.
2. Switch to the interrupt trace, not printf — see the warning above.
3. Vary the core count. -Cpus 1 versus -Cpus 4 separates a locking bug from a logic bug.
4. Suspect the build before the code. This Makefile has shipped stale kernels three separate ways; make -C kernel clean costs a minute and has ended more than one hunt. See Building.
5. Check the boundary conditions the fault sits on — a middle pointer freed as if it were a base, a higher-half alias stored as a low address, a struct field at the wrong offset.

Common root causes

Patterns that have bitten repeatedly and are worth checking first:

Pattern	Symptom
Scratch registers used before SAVE_REGS in an entry stub	Intermittent #UD or a jump to a nonsense address under interrupt load
A low kmalloc address stored where a higher-half alias is required	#PF immediately after a CR3 switch
Byte order in a network header field	Every packet silently dropped by the peer
preempt_disable() used where a spinlock is needed	Correct on one core, corrupts on four
Struct field order differing between kernel and user/syscall.h	Garbage copied across the syscall boundary
A user buffer with unfaulted lazy-heap pages	getdents/getprocs return partial data, no error
Missing cld on an entry path	Kernel string operations run backwards after a ring-3 call
Layout derived from fb_get_width() instead of the design grid	GUI elements offset or doubled in scale

See also

• Troubleshooting — symptom-first index
• Building — build flags and the stale-build traps
• Boot-Process — what each initialisation stage does
• Kernel-Data-Structures — field semantics behind the invariants above

External resources

• QEMU invocation — debug options — QEMU
• Intel SDM Vol. 3A, Ch. 6 — Interrupt and Exception Handling — exception vectors and error codes
• OSDev Wiki — Exceptions — a shorter reference for the same