Architecture

Architecture

NyxOS is a monolithic x86_64 kernel running in long mode with 4-level paging. Everything — scheduler, drivers, filesystems, TCP/IP, TLS, the window compositor, the web browser — lives in ring 0 and links into a single nyx-kernel.bin. Userspace is ring-3 ELF64 programs that reach the kernel only through the syscall instruction.

See also: Boot Process, Memory Management, Process Management, Syscalls, SMP, Security

Layer map

+-------------------------------------------------------------------+
|                       Userspace (Ring 3)                          |
|  sh.elf  init.elf  doom.elf  top.elf  edit.elf  wget.elf  nc.elf  |
|  cat wc ls ps kill grep head tail sort find cp mv stat less ...   |
|  libc.so (shared, mapped into every process)                      |
+-------------------------------------------------------------------+
|            System Call Interface — syscall/sysret, 57 calls       |
+-------------------------------------------------------------------+
|                         Kernel (Ring 0)                           |
|  +-----------+ +-----------+ +-----------+ +-------------------+  |
|  | Scheduler | |  Memory   | |    VFS    | |    Networking     |  |
|  | processes | |  paging   | |  ramdisk  | | ARP/IP/ICMP       |  |
|  | threads   | |  COW/mmap | |  EXT2     | | UDP/TCP/sockets   |  |
|  | signals   | |  heap/slab| |  proc/dev | | DHCP/DNS/HTTP     |  |
|  +-----------+ +-----------+ +-----------+ +-------------------+  |
|  +-----------+ +-----------+ +-----------+ +-------------------+  |
|  |   Crypto  | | Compositor| |    ELF    | |      Images       |  |
|  | X25519    | | 32 windows| |  loader   | | inflate/PNG       |  |
|  | P-256/384 | | 10 apps   | |  dynamic  | | BMP / GIF         |  |
|  | AES-GCM   | | Selene    | |  linker   | |                   |  |
|  | X.509/TLS | | games     | |           | |                   |  |
|  +-----------+ +-----------+ +-----------+ +-------------------+  |
|  Drivers: PS/2 kbd+mouse | PIT | RTC | ATA | VBE | SB16 | RTL8139 |
+-------------------------------------------------------------------+
|                        x86_64 Hardware                            |
|   CPU (SMP) | Local APIC | I/O APIC | PCI | PIT | CMOS | PS/2     |
+-------------------------------------------------------------------+

Source layout

All kernel code lives in kernel/, all ring-3 code in user/.

Core

File	Purpose
boot.asm	Multiboot1 + Multiboot2 headers, protected → long mode, 128 KB boot stack
kernel.c	kernel_main(), the serial/kernel shell and every builtin command
kernel.h	Types, constants, process_t, extern declarations
linker.ld	Section layout (captures -mcmodel=large .ldata/.lbss)
gdt.c idt.c isr.c irq.c	Descriptor tables, exception and interrupt entry
isr_stubs.asm stubs.asm switch.asm	Interrupt stubs, context switch, AP stubs

Memory and processes

File	Purpose
memory.c	Bitmap physical page allocator, multiboot2 memory map, page refcounts
paging.c	4-level paging, demand paging, copy-on-write, vm_handle_fault
heap.c	16 MB kernel heap (kmalloc/kfree), free-list with coalescing
slab.c	Fixed-size caches, 8 B → 1024 B
mmap.c	User VMAs — anonymous and file-backed mmap/munmap/mprotect
process.c	Process table, weighted round-robin scheduler, fork, clone, reaping
syscall.c	The 57-entry syscall dispatch and the per-process fd table
signal.c	POSIX-subset signals, delivery at return-to-ring-3, sigreturn trampoline
pipe.c	Anonymous pipes with reference-counted ends
elf.c	ELF64 loader, elf_load_image, shared-object loading
shared_libc.c	The prelinked libc.so mapped into every process; dlopen/dlsym

Filesystem and storage

File	Purpose
vfs.c	Node pool (256 inodes), mount table, /proc and /dev generation
ext2.c	Read/write EXT2 driver with a write-through sector cache
ata.c	ATA PIO driver (primary channel, 28-bit LBA)
initramfs.c	CPIO newc archive embedded as initramfs_data.h

Networking and crypto

File	Purpose
rtl8139.c ethernet.c arp.c ip.c icmp.c	Link and network layers
udp.c tcp.c net.c	Transport; net.c holds the 32-slot BSD socket layer
dhcp.c dns.c http.c	Application protocols
tls.c tls_prf.c	TLS 1.2 handshake, key schedule, record layer
curve25519.c p256.c p384.c	ECDH / ECDSA curve arithmetic
rsa.c sha256.c sha512.c aes_gcm.c	Signature verification, hashes, AEAD
der.c x509.c x509_roots.h	ASN.1 reader, chain verification, pinned trust anchors
csprng.c	HMAC_DRBG-SHA256 seeded from RDSEED/RDRAND

Graphics and applications

File	Purpose
vbe.c fb.c font.c	Bochs VBE mode setting, framebuffer primitives, 8×16 bitmap font
compositor.c	Window manager — z-order, workspaces, taskbar, Start menu, icons
*_win.c	One file per built-in app (terminal, editor, file manager, games, Selene…)
inflate.c png.c bmp.c gif.c	DEFLATE/zlib and the three image decoders
bootsplash.c login.c auth.c	Boot animation, login screen, credential store

Multi-core

File	Purpose
apic.c	Local APIC + I/O APIC, redirection entries, ISA IRQ overrides
smp.c	AP bringup (INIT–SIPI–SIPI), per-CPU state, TLB shootdown IPIs
trampoline.asm	Real-mode AP stub, embedded as a flat binary
spinlock.h	Ticket-free test-and-set spinlocks used across the kernel

Address space

The kernel occupies the top of the canonical address space; every user process gets a private PML4 whose upper half mirrors the kernel's.

0xFFFFFFFF90000000  ┬ kernel heap (16 MB)
                    │
0xFFFFFF8000000000  ┴ KERNEL_BASE — higher-half alias of all physical RAM
                      ... non-canonical gap ...
0x0000800000000000  ┬ USER_SPACE_END
0x00007FFFFFFFE000  │ top user stack page (grows down, guard page below)
                    │ mmap region
                    │ program break (lazy sbrk)
0x0000000000010000  │ ELF image load address
0x0000000000001000  ┴ USER_SPACE_MIN — page 0 left unmapped to trap NULL

Isolation. The kernel is mapped through the top PML4 entry, present in every address space so interrupts and syscalls always have their code and stacks mapped. User page tables carry no identity mapping of physical memory, so a ring-3 process cannot name kernel memory even by guessing addresses. SMEP and SMAP are enabled at boot (see Security).

Key constants

Defined in kernel/kernel.h unless noted.

Constant	Value	Meaning
KERNEL_VERSION	"5.9.86"	Reported by uname, version, /proc/version
PAGE_SIZE	4096
KERNEL_BASE	0xFFFFFF8000000000	Higher-half alias base
KERNEL_HEAP_START	0xFFFFFFFF90000000
KERNEL_HEAP_SIZE	16 MB
MAX_PROCESSES	512	Process table slots
MAX_THREADS	1024
MAX_FILES	256	Per-process fd table ceiling
MAX_MOUNTS	16
USER_STACK_TOP	0x00007FFFFFFFE000	Base of the top stack page
USER_STACK_INIT_PAGES	4	16 KB committed at load
USER_STACK_MAX_PAGES	128	512 KB demand-grow ceiling
USER_SPACE_MIN / USER_SPACE_END	0x1000 / 0x800000000000	Valid user-pointer window
MAX_CPUS (smp.h)	8
MAX_WINDOWS (compositor.h)	32
WORKSPACE_COUNT (compositor.h)	4
MAX_INODES (vfs.c)	256	VFS node pool
TCP_MAX_CONNS (tcp.h)	32
MAX_SOCKETS (net.c)	32	BSD socket slots
SLAB_MAX_OBJ (slab.h)	1024	Largest slab-served allocation

See also

• Boot Process - how this structure is brought up
• Kernel Data Structures - the structs behind the subsystems
• Memory Management - the address space in detail
• Contributing - coding standards for kernel code

External resources

• Intel SDM Vol. 3A - paging, segmentation, privilege levels
• System V AMD64 ABI - calling convention and ELF
• OSDev Wiki - Higher Half Kernel - the mapping approach used here