Building

Building and Running

NyxOS builds with a stock GCC targeting -m64 freestanding — no cross-compiler is strictly required, though x86_64-elf-gcc works too. The primary development environment is Windows with WSL: build.ps1 shells into WSL to run make, and run.ps1 launches QEMU on the Windows side.

See also: Architecture, Boot Process, Networking Stack, Version History

Prerequisites

Tool	Notes
GCC	Host gcc with -m64, or x86_64-elf-gcc
NASM	Assembles boot.asm, crt0.asm, the AP trampoline and the ISR stubs
GNU ld	Linked with -T linker.ld
grub-mkrescue + xorriso	Builds the bootable ISO (grub-pc-bin on Debian/Ubuntu)
QEMU	7.0+ recommended (qemu-system-x86_64)
Python 3	Regenerates the initramfs

On Debian/Ubuntu or WSL:

sudo apt install build-essential nasm grub-pc-bin grub-common xorriso qemu-system-x86 python3

Quick start

Windows (PowerShell)

.\build.ps1

Compiles the kernel through WSL, reports the size of kernel/nyx-kernel.bin, then generates NyxOS.iso with grub-mkrescue. Add -Clean to force a full rebuild.

.\run.ps1

Boots the ISO in QEMU. Modes and options:

Invocation	Effect
.\run.ps1	SDL display, serial log to qemu_serial.txt (default)
.\run.ps1 -Mode serial	-nographic, serial on stdio — the kernel shell
.\run.ps1 -Mode net	SDL display plus an RTL8139 NIC (-nic user,model=rtl8139)
.\run.ps1 -Mode debug	SDL display plus -d cpu_reset,int
-Cpus <n>	CPU count, default 4 (see SMP)
-Sound	Attaches an SB16 device (implied by -Mode net)

The guest gets 512 MB and -cpu qemu64. If ext2-test.img is present it is attached as -hda; NyxOS auto-mounts it at /mnt, which is also where doom1.wad must live for DOOM to start.

Linux / WSL

make -C kernel

This builds the kernel and every user/*.elf — all: depends on $(USER_ELFS) as well as nyx-kernel.bin. Then:

qemu-system-x86_64 -cdrom NyxOS.iso -m 512M -smp 4 -no-reboot -cpu qemu64 \
  -hda ext2-test.img -nic user,model=rtl8139 -serial stdio

Build system

Kernel

make -C kernel          # build kernel + user ELFs
make -C kernel clean    # remove objects, .d files and binaries

Compiler flags:

-std=gnu99 -ffreestanding -Os -Wall -Wextra
-nostdlib -nostartfiles -nodefaultlibs
-fno-stack-protector -fno-pie -m64
-mno-red-zone -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
-mcmodel=large -ffunction-sections -fdata-sections -MMD -MP -I.

-mno-red-zone is mandatory: interrupt handlers would otherwise clobber the 128-byte red zone below RSP. -mcmodel=large is what lets kernel code at 0xFFFFFF80_00000000 reference its own symbols.

The build is expected to be warning-free. If make emits warnings, that is a regression.

Stale-build protection

Three separate bugs in this Makefile once shipped stale kernels silently, and the fixes are worth knowing about:

• .DEFAULT_GOAL := all is set before -include. Each auto-generated .d file starts with a rule like kernel.o: kernel.c …; because -include sits above the all: target, the first included rule would otherwise hijack the default goal, so a bare make built only kernel.o and never relinked.
• -MMD -MP header dependencies. Every object now depends on every header it includes, so editing a macro in tcp.h rebuilds tcp.o. Previously only kernel.h was tracked, and a header-only change left stale objects — which masked a real capacity fix as a phantom "Heisenbug".
• linker.ld is a prerequisite of nyx-kernel.bin. Editing the section layout now forces a relink.

If you suspect a stale build, make -C kernel clean is the reliable answer.

Userspace programs

User ELFs are built by the same Makefile from user/. Each is linked as:

ld -nostdlib -m elf_x86_64 -e _start -Ttext 0x10000 \
   -o prog.elf crt0.o --just-symbols=libc.so prog.o

--just-symbols=libc.so resolves libc symbols against the prelinked shared library without copying its code in — libc exists once in memory and is mapped into every process. A userspace program compiles with:

-std=gnu99 -Os -ffreestanding -nostdlib -m64 -mno-red-zone -I../user

See Userspace for the ABI and the available library surface.

Initramfs

The root filesystem is a CPIO (newc) archive embedded in the kernel as a C byte array. Regenerate it after adding or changing files:

python3 tools/mkinitramfs.py kernel/initramfs_data.h c

ISO

Both build.ps1 and run.ps1 generate iso/boot/grub/grub.cfg and call:

grub-mkrescue -o NyxOS.iso iso/

Troubleshooting

Symptom	Cause / fix
gcc: command not found	Install GCC, or set CC in kernel/Makefile
nasm: command not found	sudo apt install nasm
grub-mkrescue: not found	sudo apt install grub-pc-bin grub-common xorriso
ISO builds but QEMU shows the GRUB rescue prompt	grub-mkrescue needs xorriso; check its stderr, which build.ps1 suppresses
Changes don't take effect	Stale build — make -C kernel clean and rebuild
Triple fault on boot	Run -Mode debug (-d cpu_reset,int) and read the last interrupt vector; usual causes are bad page tables, an unloaded IDT, or a boot-stack overflow
Desktop never appears	VBE mode set failed; check the serial log for VBE mode set failed — the kernel falls back to the text shell
DOOM exits immediately	/mnt/doom1.wad is missing; attach ext2-test.img
No network	Boot with -Mode net (or add -nic user,model=rtl8139); check ifconfig for a DHCP lease
qemu-system-x86_64 not found	run.ps1 probes the usual install paths; add QEMU to PATH if it lives elsewhere

Debugging

• Serial log. In gui and net modes everything the kernel prints goes to qemu_serial.txt. This is the first place to look.
• Interrupt trace. -Mode debug adds -d cpu_reset,int. Unlike serial printf, this does not perturb timing, which is how several race conditions in this codebase were actually found.
• Kernel shell. -Mode serial gives an interactive kernel shell over stdio with mem, ps, cpus, hexdump and the whole self-test suite. See Shell.
• Self-tests. The kernel ships known-answer tests for most cryptographic and image code — x25519test, gcmtest, p256test, p384test, rsatest, sha512test, chaintest, csprngtest, deflatetest, pngtest, bmptest, giftest — plus cowtest, tlbtest, smpstress and mtdemo for the kernel core.