SMP

SMP — Symmetric Multi-Processing

NyxOS boots up to 8 cores and runs both kernel threads and ring-3 user processes on the application processors, with real spinlocks over shared kernel state and TLB shootdown IPIs on the paths that remove a mapping.

See also: Architecture, Process-Management, Memory-Management, Drivers

Status

Max CPUs	8 (MAX_CPUS in smp.h)
Detection	CPUID leaf 1 / leaf 0x80000008
Interrupt mode	Local APIC + I/O APIC
AP kernel threads	✅ Running, per-CPU scheduler
AP user processes	✅ Ring 3 on application processors
Spinlocks	✅ Physical allocator, heap, slab, VFS node pool, mount table, keyboard, network stack, EXT2
TLB shootdown	✅ On munmap, mprotect, and the COW remap
Process balancing	Opt-in (smpbalance); default placement is unchanged

Test with -smp 4 — run.ps1 passes -Cpus 4 by default.

Bringup

1. smp_init() records the BSP as CPU 0 and detects the AP count.
2. For each AP: INIT IPI → wait 10 ms → SIPI pointing at the trampoline at 0x8000.
3. The AP starts in real mode and climbs to long mode through trampoline.asm.
4. It calls ap_main(cpu_id), loads the shared GDT/IDT, records its own APIC id via CPUID, and enters the scheduler.

Trampoline (trampoline.asm)

A position-independent real-mode stub assembled as a flat binary and embedded into the kernel with ld -r -b binary. It sets up segments, enables A20, enters protected mode, then long mode, and jumps to ap_main.

Per-CPU state

typedef struct {
    uint32_t apic_id;        // assigned by the BSP
    uint32_t apic_id_self;   // read by the core itself via CPUID
    uint32_t cpu_number;
    uint64_t stack_base;
    uint64_t stack_top;
    volatile int started;
    volatile int running;
} cpu_info_t;

cpu_self() resolves the running core from the initial APIC id in CPUID(1).EBX[31:24]. It reads that from an instruction, not from memory, which makes it valid before the LAPIC mapping is reachable and safe to call from inside an interrupt handler.

Several things that used to be global are now per-CPU:

State	Why
user_rsp, user_cr3, kernel_rsp, syscall_frame_ptr	Two cores entering the kernel at once would otherwise overwrite each other's saved ring-3 state. Reached through GS, because syscall_entry has no spare register
preempt_count	A global count meant one core's critical section suppressed another core's preemption
IST stacks (double fault, NMI)	A fault on one core must not scribble on another core's exception stack. One 8 KB DF stack + one 8 KB NMI stack per online core
TSS descriptor	A TSS may be busy on only one core at a time — which is exactly why APs previously skipped ltr, and why a ring-3 → ring-0 entry now lands on the kernel stack of the process that core is running

get_current_process() resolves per-CPU. It used to answer with the BSP's task for anything running on an AP, which was the root cause of two separate multi-core hangs.

Scheduling on the APs

The AP timer is a real context-switching ISR feeding a per-CPU scheduler. process_t.sched_cpu pins a task to one core, and that pin is the entire mutual-exclusion argument — no other core will even consider a task pinned elsewhere. The AP scheduler needed no new logic of its own; it calls the same sched_target() the BSP uses, because every slot that function touches became per-CPU during bringup.

smpthreads [secs] wakes one worker per AP and reports the iteration count each core actually executed — measured at 30.9M / 28.0M / 25.7M on CPUs 1/2/3 while the sleeping BSP counted zero.

smpuser [n] spreads ring-3 processes round-robin across cores.

Locking

Spinlocks (spinlock.h), taken with interrupts disabled, protect:

• The physical page allocator, kernel heap, and slab caches
• The VFS node pool and mount table
• The keyboard input path and terminal line discipline
• The network stack (one net_lock)
• The EXT2 driver

preempt_disable() still exists but is a different tool: it stops a context switch on the local core and says nothing about another one. Anything a second core can reach concurrently needs a spinlock.

smpstress [secs] hammers the allocators from every core at once — roughly 900k operations with zero integrity failures and a machine-wide free-page count that returns exactly balanced.

TLB shootdown

tlb_shootdown() broadcasts vector 0x41 and waits for every core to acknowledge. It is called from every path that removes or restricts a mapping: munmap, mprotect, and the copy-on-write remap (which replaces a translation a sibling thread may still be holding).

Two design points matter:

• The shootdown lock is taken without disabling interrupts, so two concurrent shootdowns cannot deadlock waiting on each other.
• The callers run with interrupts disabled, so the shootdown enables them for its duration and restores the caller's flag afterwards. Without that, two cores unmapping simultaneously would each wait forever on the other.

tlbtest proves it the only way that means anything: it first shows CPU 1 reading a stale value after a remap with no IPI, then the fresh value once the IPI is sent.

Inspecting

> cpus
SMP: 4 CPU(s) online (max 8)
CPU  ROLE  APIC  SELF  STATE
0    BSP   0     0     online
1    AP    1     1     online
2    AP    2     2     online
3    AP    3     3     online

Command	Purpose
cpus	Core list with per-CPU tick counters
smpstress [secs]	Allocator contention test across all cores
smpthreads [secs]	One kernel thread per AP, reporting real per-core throughput
smpuser [n]	Spread user processes across cores
smpbalance [on|off]	Opt-in automatic placement of new processes and threads
tlbtest	Cross-CPU TLB shootdown proof

Known limits

• Thread groups stay pinned to one core, because they share page tables.
• Automatic balancing is opt-in. It is safe for long-lived processes; heavy fork/exec churn with balancing enabled is not a supported configuration.
• No IRQ affinity. Device interrupts are delivered to the BSP.

See also

• Process-Management - the scheduler the APs run
• Memory-Management - why shootdowns are needed
• Drivers - the APIC and the per-core LAPIC timer
• Debugging - separating a locking bug from a logic bug

External resources

• Intel SDM Vol. 3A, Ch. 8 - Multiple-Processor Management - Intel
• OSDev Wiki - Symmetric Multiprocessing - OSDev
• OSDev Wiki - TLB Shootdown - OSDev