SharpMetal v2.0.0: Hardened VMM, Frontiers 1–5 & POSIX Subsystem

SharpMetal v2.0.0: The Microkernel Frontier

SharpMetal is an experimental, bare-metal x86_64 microkernel written in 100% pure C# and compiled to zero-dependency native code using .NET 10 Native AOT. Operating at Ring 0 with a capability-based security model inspired by seL4, it features zero garbage-collection overhead in core IPC/scheduler paths and executes within a Higher-Half Direct Map (HHDM) memory layout.

Version v2.0.0 introduces the Frontiers 1–5 architectural hardening milestone: a hardened VMM with demand paging and COW, dynamic ELF/PIE execution, a full Layer 2–4 network stack, a native USB 3.0 xHCI host driver with HID support, and a userland POSIX compatibility layer.

---

What's New in v2.0.0

1. Frontier 1: VMM & SMP Concurrency Hardening

• Lock-Free COW Refcounting: PhysicalFrameRefcount converted to atomic Compare-And-Swap (CAS) loops, eliminating spinlock contention during high-frequency Copy-On-Write page sharing and unsharing.
• Inter-Processor TLB Shootdown: SmpTlbShootdown integrated across all mutation paths—both during frame duplication and when single-reference pages are upgraded back to writable mode.
• Canonical Address Boundary Enforcement: The fault handler strictly enforces the < 0x0000_8000_0000_0000 bound, immediately terminating unprivileged processes that attempt to probe the higher-half direct map (HHDM).

2. Frontier 2: Dynamic ELF/PIE Relocation & Execution

• Userland.PieLoader gains a zero-alloc ELF64 parser (Elf64_Ehdr/Phdr/Dyn/Rela) with PT_LOAD segments registered as demand-paged, file-backed VMAs.
• R_X86_64_RELATIVE relocations applied against an ASLR base; kernel SysSpawnElf synthesizes the System V initial stack (argc/argv/envp + AT_PHDR/AT_ENTRY/AT_RANDOM auxv) and drops to Ring 3.

3. Frontier 3: VirtIO Network Driver & Microkernel IP Stack (net.stack)

• Layer 2/3/4: Ethernet framing, ARP cache with expiration, IPv4 routing with checksum calculation, ICMP echo responder, UDP sockets, and a full TCP state machine (CLOSED through TIME_WAIT) with RTO retransmission.
• POSIX-like Socket RPC: socket, bind, listen, accept, connect, send, recv, and close exposed over the capability-secured INetworkService endpoint.
• VirtIO-Net: Split RX/TX virtqueues with zero-alloc DMA replenishment and interrupt-driven RX via sys_recv_any.

4. Frontier 4: USB 3.0 xHCI Controller & HID Drivers (bus.xhci)

• Native xHCI Subsystem: Full hardware lifecycle management—DCBAA, Command/Event/Transfer rings, ERST, per-ring cycle-bit tracking, doorbell arrays, and scratchpad provisioning.
• USB HID Boot Protocol: Keyboard and mouse enumeration, 8-byte boot report translation via HID usage tables, and keys injected into input.hid via the InjectKey RPC.
• Fail-Safe Legacy Handoff: The USBLEGSUP ownership handshake is timeout-bounded; on timeout, the driver cleanly aborts without disabling BIOS legacy emulation, keeping the PS/2 input path functional. A bare-metal safety gate spawns bus.xhci only when xhci_native.flag is present in the initrd or a hypervisor is detected, keeping PS/2 authoritative on physical hardware unless explicitly opted in.

5. Frontier 5: POSIX / Libc & WASI Compatibility Layer (Microkernel.Posix)

• Syscall ABI Emulation: Per-thread Linux ABI mode in the kernel dispatch table translates Linux x86_64 syscall numbers (read, write, open, close, mmap, brk, socket, exit) into SharpMetal capability RPCs.
• FD Multiplexer: FDs 0/1/2 route to the console/input service; FDs >= 3 route to FAT32/VFS handles or network sockets.
• POSIX Host Runner: posix_runner hosts statically linked PIE binaries in isolated Ring 3 capability domains.

6. Freestanding Native AOT Runtime Stubs

• Standalone runtime stubs (RhpAssignRef, RhpNewFast, RhpPInvoke, __security_cookie) implemented across all modular userland drivers, stripping away unnecessary CoreLib dependencies.

7. Build Pipeline & Concurrency Performance

• Make-DiskImage.sh packages 12 initrd payloads (11 zero-alloc service binaries plus the xHCI native opt-in flag) into disk.img.
• Removed -maxcpucount:1 throttling, unlocking full multi-core Roslyn and NASM compilation with safe sequential project ordering.
• Added Rebuild-KernelOnly.sh for rapid kernel turnaround.

---

Quick Start & Installation

Running via QEMU (Recommended)

Ensure QEMU and OVMF UEFI firmware are installed, then execute:

qemu-system-x86_64 \
    -M q35 \
    -cpu host -enable-kvm \
    -m 2G \
    -smp 4 \
    -bios /usr/share/ovmf/OVMF.fd \
    -drive file=disk.img,format=raw,if=none,id=nvm \
    -device nvme,serial=deadbeef,drive=nvm \
    -netdev user,id=net0,hostfwd=udp::8080-:8080 \
    -device virtio-net-pci,netdev=net0 \
    -device qemu-xhci,id=xhci \
    -device usb-kbd,bus=xhci.0 \
    -serial stdio

(Note: On systems without /usr/share/ovmf/OVMF.fd, install ovmf or edk2-ovmf via your system package manager).

Flashing to Physical USB Drive (Bare-Metal)

Identify your target USB drive (lsblk) and write the raw GPT disk image:

# Replace /dev/sdX with your actual USB target drive
sudo dd if=disk.img of=/dev/sdX bs=4M status=progress oflag=sync

Reboot into UEFI setup, disable Secure Boot, and select the USB drive. On bare metal, the PS/2/legacy keyboard path remains authoritative unless xhci_native.flag is added to the initrd.

---

Included Release Assets

• disk.img: Complete 64MB GPT disk image containing the EFI system partition (FAT32), direct UEFI application boot, the SharpMetal microkernel binary, and the 12-payload INITRD.IMG.
• nvme.img: 64MB NVMe storage disk image pre-formatted with FAT32 containing userland test binaries (/bin/test.pie, /bin/posix_test, /bin/cat.pie, /bin/hello.pie) and sample assets (/HELLO.TXT).
• BOOTX64.EFI: Standalone x86_64 UEFI Native AOT executable.

SharpMetal v1.1.0

SharpMetal v1.1.0 — Architecture Hardening

Hardened bare-metal C# microkernel running on native AOT with verified Layer 7 Ring 3 userland transition, serialized SMP initialization, and fault-tolerant storage/network device drivers.

---

1. Release Overview & Changelog

Layer 7 Ring 3 Transition & Privilege Drops

• General Protection Fault Fix (#GP, Vector 0x0D): Resolved privilege mismatch panics during iretq by assigning valid Ring 3 segment selectors (0x23 for User 64-bit CS with RPL=3, 0x1B for User DS/SS with RPL=3) in UserTransition.asm and Gdt.cs.
• Canonical Userland RIP Mapping: Updated ObjectSynthesizer.cs and RoottaskEntry.asm so the synthesized Root Task thread points to the userland PIE base in lower canonical memory rather than higher-half kernel return addresses (0xFFFF8001...).
• Context Preservation: Retained IA32_GS_BASE per-CPU scratch mappings without executing ungrounded swapgs instructions during transitions.

Core SMP Real-Mode Bootstrap

• Serialized AP Trampoline: Implemented an atomic 16-bit spinlock (trampoline_lock) in ApTrampoline.asm released once an AP transitions to long mode (long64_entry), preventing AP memory bus stomping on shared low-memory pages (0x8000).
• Per-Core Stack Safety: SmpBootstrap.cs assigns dedicated 16 KiB stacks per core indexed by APIC ID (supporting topologies up to 16 cores) and guards against double-allocating live or booting processors.
• Null-Stack Trap: Added an explicit null check in Entry.asm (ApEntry64) to force an AP halt (hlt) rather than jumping on unassigned stack pointers.

NVMe Storage Engine

• Dynamic PRP Lists: Introduced a dynamic Page Request Principle (PRP) list arena at DMA address 0x71005000 in NvmeDriver.cs, allowing multi-block transfers exceeding 8 KiB (up to 2 MiB per transaction) without sector truncation.
• Doorbell & Head Desynchronization Fix: Placed volatile memory barriers between SQE population and doorbell writes, and prevented CQ head advancement when polling cycles hit completion timeouts.

Subsystem Resiliency & IPC

• Interrupt Stack Table (IST) Hardening: Assigned dedicated 16 KiB per-CPU IST1 stacks in Idt.cs across NMI (#2), Double Fault (#8), and Machine Check (#18) handlers to eliminate reentrancy races across GS-base structures.
• Capability & Endpoint Sweep: Added Endpoint.PurgeThread() to sweep queued send/receive rendezvous states upon thread termination while preventing capability leaks in CapabilityDerivationTree.cs.
• FAT32 Cluster Sanitization: Applied & 0x0FFFFFFF masking across directory cluster entries and root cluster fields in Fat32Driver.cs to ignore upper reserved bits.
• VirtIO-Net Synchronization: Added acquire-side memory barriers across ring index polling loops in VirtioNetDriver.cs and hardened index calculations against modulo-65536 rollover.

---

2. Prerequisites & Environment Setup

Ensure the following toolchains are installed on the host system:

• .NET SDK (matching the repo global configuration)
• NASM (Netwide Assembler) for low-level x86_64 trampolines
• QEMU x86_64 Emulator (qemu-system-x86_64) with OVMF firmware
• Python 3 (for the automated verification harness)
• Standard GNU Utilities (bash, mtools, xorriso)

---

3. Build & Packaging Instructions

Execute these commands from the repository root.

Step 1: Reassemble Low-Level Trampolines

nasm -f bin src/kernel/Arch/x86_64/Assembly/ApTrampoline.asm -o src/kernel/Arch/x86_64/Assembly/ApTrampoline.bin

Compiles the raw 16-bit real-mode bootstrap binary embedded directly into the kernel image.

Step 2: Compile the Microkernel Solution

dotnet build Microkernel.sln -c Release

Runs Native AOT compilation across all kernel modules, runtime libraries, drivers, and userland applications with zero warnings and zero errors.

Step 3: Pack Initrd & Build Bootable Disk Image

bash build/scripts/Make-DiskImage.sh

Packages userland servers into the initrd and formats build/disk.img containing the EFI System Partition and FAT32 layout.

---

4. Running the Kernel

Interactive Graphical Mode (GUI Framebuffer)

Launches the microkernel with the GOP framebuffer display server, PS/2 mouse/keyboard, NVMe storage, and VirtIO-net adapter attached:

bash build/scripts/Run-Qemu.sh

Headless Console Mode (Serial Debugging)

Routes COM1 early serial diagnostic logs directly to the active terminal stdout:

bash build/scripts/Run-Qemu.sh --headless

Automated Milestone Test Harness

Runs headless verification against all 10 architectural boot phases and asserts serial milestones:

python3 build/scripts/Test-Harness.py

SharpMetal v1.0.2: Paging Reclamation, APIC Calibration & Dynamic Exec

SharpMetal Microkernel Architectural Hardening & Dynamic Execution (v1.0.2)

Highlights & Changes

• MMIO-Safe Address Space Teardown: Added iterative page table destruction (DestroyAddressSpace) with PageFrameAllocator.IsRam validation to prevent MMIO and framebuffer ranges from being corrupted during process exits.
• VirtIO Virtqueue Fixes: Resolved 16-bit ring index wraparound edge cases on avail->idx and used->idx, and enforced write memory barriers before queue doorbells.
• APIC Timer Hardware Calibration: Calibrated Local APIC timer ticks dynamically against PIT Channel 2 (11,932 divisor / 10ms window) for reliable multi-core scheduling.
• Dynamic Exec via Syscall 0x0B: Implemented sys_spawn and userland PieLoader to relocate and execute freestanding PIE binaries dynamically.
• Runtime & Toolchain Alignment: Added missing ThrowFileNotFoundException runtime stub to MiniCoreLib and resolved reference assembly paths for Native AOT ILCompiler.

Quick Start (Booting under QEMU)

1. Download pre-built disk image from release assets

   curl -LO [https://github.com/Sadik00789/SharpMetal/releases/download/v1.0.2/disk.img](https://github.com/Sadik00789/SharpMetal/releases/download/v1.0.2/disk.img)

2. Create backing image for NVMe benchmark storage

   qemu-img create -f raw nvme.img 64M

3. Launch QEMU (with 4 cores, UEFI firmware, NVMe, and VirtIO-Net)

   qemu-system-x86_64 -machine q35 -cpu max -smp 4 -m 1G \
     -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
     -drive file=disk.img,format=raw \
     -drive file=nvme.img,format=raw,if=none,id=nvm \
     -device nvme,serial=nvme01,drive=nvm \
     -netdev user,id=net0 -device virtio-net-pci,netdev=net0

SharpMetal v1.0.1 — Dynamic SMP Multiprocessing, Hardware TSS Isolation & Higher-Half Descriptors

SharpMetal v1.0.1: Dynamic SMP Multiprocessing, Hardware TSS Isolation & Higher-Half Descriptors

Release 1.0.1 introduces Dynamic Symmetric Multiprocessing (SMP) to SharpMetal (supporting dynamic enumeration of up to 16 cores via ACPI MADT, verified with 4 vCPUs in CI), along with architectural hardening for descriptor table virtualization, per-core hardware Task State Segment (TSS) isolation, and atomic SMP serial logging.

What's New & Architectural Updates

• Dynamic Symmetric Multiprocessing (SMP):

• Dynamic ACPI MADT Core Discovery: Automatically detects core topology, Local APIC IDs, and processor UIDs from firmware ACPI tables, dynamically scaling across available system cores (supporting up to 16 cores).
• AP Bootstrap Engine: Stages a 16-bit real-mode trampoline at physical address 0x0000_8000, issuing hardware INIT-SIPI-SIPI sequences via the Local APIC to bring all Application Processors online directly into 64-bit Long Mode.
• Broadcast IPI TLB Shootdown: Synchronizes page table modifications across all active cores using vector 0xFD inter-processor interrupts with atomic acknowledgment bitmasks.
• Multi-Core Preemptive MLFQ Scheduler: Distributes thread execution across active cores with per-core idle loops and concurrent queue dispatching.

• Strict Higher-Half Descriptor Addressing:

• Ensures GDT, IDT, and TSS base structures and PerCpuData pointers (IA32_GS_BASE) reside strictly in canonical higher-half virtual memory (Hhdm.Base).
• Guarantees faultless Ring 3 userland operation where low identity memory (PML4[0]) is unmapped.

• Per-Core Hardware TSS & Stack Isolation:

• Each core maintains an isolated Task State Segment and dedicated 16 KiB interrupt stack.
• TaskStateSegment.SetRsp0 exclusively updates the calling core's TSS.Rsp0 and KernelRsp, eliminating cross-core interrupt stack corruption during context switches.
• Segment reload operations (ReloadSegments) hardened to preserve IA32_GS_BASE.

• Atomic Whole-Message Serial Logging:

• Replaced character-by-character UART logging in SysLog with whole-message atomic locking via SpinLockWithIrqSave, eliminating inter-core character interleaving during concurrent userland server startup.

Verification & CI

• Verified under headless QEMU with 4 vCPUs (-smp 4) and OVMF firmware.
• All 10 boot milestones verified with zero unhandled faults, zero panics, and clean exit code 0:

• [SMP] 4 cores synchronized and operational
• [PASS] Concurrent zero-alloc physical frame stress test succeeded
• [PASS] Broadcast IPI TLB shootdown verified across all active cores
• [ROOTTASK] Initial root CNode initialized
• [PCI] Scanning PCIe ECAM bus topology
• [PCI] Found Host Bridge
• [NVME] Controller initialized
• [NVME] Block I/O benchmark passed
• [VIRTIO-NET] Modern PCI VirtIO Network device detected
• [SHELL] SharpMetal Bare-Metal Shell online

---

Quick Start (Booting under QEMU)

# 1. Download pre-built disk image from release assets
curl -LO https://github.com/Sadik00789/SharpMetal/releases/download/v1.0.1/disk.img

# 2. Create backing image for NVMe benchmark storage
qemu-img create -f raw nvme.img 64M

# 3. Launch QEMU (e.g. with 4 cores, UEFI firmware, NVMe, and VirtIO-Net)
qemu-system-x86_64 -machine q35 -cpu max -smp 4 -m 1G \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive file=disk.img,format=raw \
  -drive file=nvme.img,format=raw,if=none,id=nvm \
  -device nvme,serial=nvme01,drive=nvm \
  -netdev user,id=net0 -device virtio-net-pci,netdev=net0

SharpMetal v1.0.0 — Freestanding C# Bare-Metal Microkernel

SharpMetal v1.0.0: 12-Layer Bare-Metal x86-64 Microkernel in Freestanding C#

Initial production release of SharpMetal, an seL4-inspired microkernel operating system written in freestanding C# (.NET 9 Native AOT) for modern x86-64 hardware with zero dependencies on CoreCLR, glibc, or external bootloaders.

Architectural Highlights

• Freestanding Native AOT: Zero CoreCLR/Mono runtime dependencies; executes directly on bare metal using a custom MiniCoreLib with unmanaged entry points and zero GC statics.
• seL4-Style Capability Model & CDT: Guarded CSpace tables with a zero-alloc Capability Derivation Tree (CDT) enforcing recursive capability revocation, synchronous PTE cleardown, and TLB shootdown (invlpg).
• Synchronous IPC & Timeslice Donation: Fast-path register-based rendezvous IPC with timeslice donation, 64-bit atomic notifications, and unified dual-wait reactors (sys_recv_any).
• Interrupt-Driven Ring 3 Drivers: Isolated userland drivers for NVMe DMA block storage and modern VirtIO-Net driven via explicit PCI MSI/MSI-X vectors (0x30 and 0x31) instead of polling.
• Dynamic ZeroAlloc Arena: Chunk-linked NativeArena expansion (ArenaChunk) in Userland.Runtime.ZeroAlloc mapped at high DMA aperture 0x0000_7000_0000_0000UL to prevent driver starvation.
• Persistent Filesystem & VFS: Complete FAT32 cluster chain parser with System.IO.File streaming abstractions over shared DMA memory.
• AVX2 Accelerated Compositor: 256-bit SIMD software compositor blitting to UEFI GOP linear video memory using Write-Combining cache attributes and dirty-region clipping.
• Fault-Tolerant Supervisor: Watchdog-based crash interception, capability revocation, and automated PCIe Function-Level Reset (FLR) driver reincarnation.
• Interactive Graphic Terminal Shell: Micro-GC userland shell with command history ring buffering, PSF2 font rendering, hardware benchmarking tools, and ACPI shutdown support.

Verification & CI

• Automated streaming test harness validated under headless QEMU with OVMF firmware, asserting sequential milestones across all 12 layers (CSpace, PCI ECAM, NVMe canary write/read, VirtIO TX ring, FAT32 VFS, and Shell) with clean exit code 0.

---

Quick Start (Booting under QEMU)

# 1. Download pre-built disk image from release assets
curl -LO https://github.com/sadik00789/SharpMetal/releases/download/v1.0.0/disk.img

# 2. Create backing image for NVMe benchmark storage
qemu-img create -f raw nvme.img 64M

# 3. Launch QEMU with UEFI firmware, NVMe, and VirtIO-Net
qemu-system-x86_64 -machine q35 -cpu max -m 1G \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive file=disk.img,format=raw \
  -drive file=nvme.img,format=raw,if=none,id=nvm \
  -device nvme,serial=nvme01,drive=nvm \
  -netdev user,id=net0 -device virtio-net-pci,netdev=net0