Skip to content

Memory Management

Jump to bottom
kazah-png edited this page Aug 6, 2026 · 7 revisions

Memory Management

Four layers: a bitmap physical page allocator, 4-level paging with demand faulting, a kernel heap, and slab caches for small objects. On top of them sits the userspace VM — lazy sbrk, mmap VMAs, copy-on-write fork, and demand-grown stacks.

See also: Architecture, Boot-Process, Process-Management, SMP, Security

Physical allocator (memory.c)

A bitmap with one bit per 4 KB frame. alloc_page() returns a physical address, free_page() releases one.

Firmware-aware. The allocator is given the Multiboot 2 memory map — every {base, length, type} entry the bootloader reported — rather than assuming a flat region of RAM. Regions the firmware did not mark as available are never handed out. This matters: handing out a reserved page produced a memory-corruption bug that took many releases to pin down, because the symptom (a corrupted shell pipeline) was nowhere near the cause.

Reference counting. Each frame carries a share count. page_incref() bumps it when a frame becomes shared — by fork's copy-on-write clone, or by the shared libc mapped into every process — and free_page() only actually frees at the last reference. Without this, one process exiting would unmap pages still live in its sibling.

Note

Allocator hardening (v6.4.83–84). The physical allocator now rejects a non-page-aligned free, a silent double-free, and an incref of an already-free frame. The kernel heap was hardened alongside it: krealloc keeps the original block on OOM (rather than leaking it), and kfree rejects a header with an unknown magic instead of corrupting the free-list. Each is a footgun removed, not a new guarantee.

Allocation is protected by a spinlock taken with interrupts disabled, not by preempt_disable(). The distinction is the whole point on a multi-core machine: disabling preemption stops a context switch on the local core and means nothing to another one. The smpstress command hammers the allocator from every core at once to prove it holds.

Paging (paging.c)

Standard x86_64 4-level translation: PML4 → PDPT → PD → PT → 4 KB page. The kernel lives in the top PML4 entry and is mirrored into every address space, so an interrupt or syscall taken in any process always finds its code and stacks mapped. All physical RAM is aliased at KERNEL_BASE (0xFFFFFF8000000000) — the identity map covers all detected RAM, not a fixed 64 MB window as it once did.

Software PTE bits

The x86 page-table format leaves bits 9–11 available to the OS. NyxOS uses them to record why a page is absent or restricted:

Bit Name Meaning
9 PTE_DEMAND Not yet allocated; supply a zeroed frame on first touch
10 PTE_COW Shared read-only; make a private copy on write

vm_handle_fault

The page-fault handler is the single place where lazy memory becomes real. It resolves:

  • Demand-zero pages — a PTE_DEMAND entry becomes a fresh zeroed frame
  • Copy-on-write — a write to a PTE_COW page allocates a private copy and drops the shared reference
  • Lazy sbrk — a fault inside [heap_start, program_break) materialises a heap page
  • File-backed mmap — the faulting page's slice is copied out of the VMA's snapshot buffer
  • Stack growth — a fault just below the current stack extends it (see below)

Anything else in the user half is a genuine fault, and becomes a SIGSEGV delivered to the process rather than a kernel panic (see Process-Management).

Control-register state

  • CR0.WP — supervisor writes to read-only pages fault. Required for copy-on-write to work when the kernel writes into a user page.
  • NX — user stack, heap and data pages are mapped non-executable.
  • SMEP / SMAP — enabled at boot; see Security.

Warning

get_phys_addr and the NX bit (v5.9.105). The virtual→physical translator masked page-frame bits with & ~0xFFF, which clears only the low 12 bits and keeps bit 63 (NX). On any NX page — every data page — it therefore returned physaddr | (1<<63), a non-canonical garbage pointer. It was not triggered in practice only because its sole callers happened to use non-NX pages. All frame extraction now masks bits [51:12] through a single canonical PHYS_ADDR_MASK (three duplicate mask macros were consolidated at the same time).

User address space

Demand-grown stacks

User stacks start at USER_STACK_TOP (0x00007FFFFFFFE000) and grow down.
Committed at load 4 pages (16 KB)
Ceiling 128 pages (512 KB)
Guard page The page below USER_STACK_LOW is never mapped

A fault in the growth window extends the stack by a page; a fault below the guard page is a real stack overflow and kills the process. Before this, every process pre-committed a fixed 64 KB whether it needed it or not.

Lazy sbrk

SYS_SBRK moves program_break without allocating anything. Pages appear on first write. A malloc(8000) therefore costs only the pages the program actually touches, and the break is inherited across fork.

mmap (mmap.c)

VMAs are recorded in a per-thread-group table; mappings live in [4 GiB, 112 TiB), clear of both the heap and the stack.

Call Behaviour
mmap anonymous MAP_PRIVATE demand-zero pages; prot honoured (writable only with PROT_WRITE, NX unless PROT_EXEC)
mmap file-backed The file is snapshotted into a per-VMA kernel buffer at the given offset; faulting pages copy their slice out of it
mprotect Rewrites the flags of present pages and the VMA's recorded prot. A partial-range call splits the VMA so the change is precise
munmap Frees present pages (refcount-aware) and drops the VMAs; a partial-range call splits the VMA rather than silently doing nothing

Mappings are inherited copy-on-write across fork (file buffers are deep-copied), and dropped by execve.

Every path that removes or restricts a mapping — munmap, mprotect, the COW remap — issues a TLB shootdown IPI so no other core keeps a stale translation. See SMP.

Copy-on-write fork

clone_page_directory_cow() walks the user half of the parent's PML4:

  • Writable leaf pages are downgraded to read-only + PTE_COW in both parent and child, and the frame's refcount is bumped
  • Read-only pages (program text) are shared as-is
  • The first write by either side faults, vm_handle_fault allocates a private copy, and the shared reference is dropped

The child returns 0 from fork(), the parent returns the child's pid, and both continue in the round-robin.

Threads are different. clone(CLONE_VM) deliberately does not copy the address space — the new task shares it. See Process-Management.

Kernel heap (heap.c)

A 16 MB heap at KERNEL_HEAP_START with an address-ordered first-fit free-list, header/footer metadata and block coalescing.

kmalloc routes small requests to the slab allocator and falls back to the heap when no slab class fits. Allocations carry one of two header magics (_SLAB / _HEAP) so kfree routes correctly regardless of size. This is not a detail: for several releases kmalloc sent everything ≤ 1024 B to a slab that only had classes up to 512 B, so every allocation between 505 and 1016 bytes silently returned NULL — which showed up as files that wrote successfully but came back empty.

Note

Fragmentation fix (v5.9.110). heap_free coalesced only forward and only once, so two physically-adjacent blocks freed while a block between them was still in use stayed split forever — over a long uptime this fragmented the heap until a large request (a 64 KB HTTP response, an image plane) failed despite ample free space. heap_alloc now coalesces each free block with all following free blocks as it walks the address-ordered list, so space freed in any order is reclaimed on the next allocation.

Slab allocator (slab.c)

Fixed-size object caches at 8, 16, 32, 64, 128, 256, 512 and 1024 bytes (SLAB_MAX_OBJ). Anything larger goes to the heap. Like the physical allocator, the slab caches are spinlock-protected for SMP safety.

Inspecting memory at runtime

Where What
mem (kernel shell) Physical page usage and heap statistics
free (ring 3) Reads /proc/meminfo
pmap [pid] (ring 3) Reads /proc/<pid>/maps — the process's mapped regions
cowtest (kernel shell) Self-test for demand paging and copy-on-write
tlbtest (kernel shell) Proves cross-CPU TLB shootdown works, by first showing a stale read without it

See also

External resources

NyxOS v6.4.363 · GPL v2 · GitHub · uselessalter on Discord · nyxos@inbox.lv

NyxOS Wiki

Getting started

Kernel

Storage & network

Graphics & apps

Userspace

HOWTO

Reference

Project

Clone this wiki locally