Production Readiness Roadmap

Production Readiness Roadmap

Purpose

This page defines what OSAI needs to become useful for production deployment. The QEMU target is complete (milestones 0-71), but the OS is still a single-boot, cooperative, self-test-oriented prototype. This roadmap covers the gap between "QEMU complete" and "production useful."

Items are ordered by priority within each tier. Higher tiers block lower tiers.

Tier 1 is complete as of commit 8083940 (2025-06-16). All 5 core OS items implemented: preemptive scheduler, ELF loader with address space isolation, persistent filesystem, TCP/IP stack, and SSH server. 53 files changed, ~3,900 lines added.

Tier 2 is complete as of 2025-06-16. All 5 operability items implemented: RTC/wall time, persistent logging with ring buffer, crash recovery with backoff, package delivery with SHA-256 verification, and hardware watchdog with panic recovery. 8 new files, 10 modified files.

Status Legend

Marker	Meaning
⬜ Not started	Planned but no implementation yet.
🔵 In progress	Active engineering work.
✅ Done	Implemented and tested.
⚠️ Blocked	Waiting on prerequisite design, code, hardware, or tooling.

Contents

• Tier 1: Core OS Functionality
• Tier 2: Operability
• Tier 3: AI Agent Capability
• Tier 4: Hardware Readiness
• Tier 5: Ecosystem
• Effort Estimates

---

Tier 1: Core OS Functionality

Without these, nothing real can run. These are the critical-path items.

#	Item	Description	Status
1	Real preemptive scheduler with concurrent execution	Round-robin scheduler on CPU 0 with GICv3 PPI timer interrupts (INTID 30). Full context save/restore (34 registers). CPUs 1-3 remain leased AI cores with no-migration guarantee. scheduler_tick() drives preemption and address space switching.	✅ Done
2	Real ELF loader with proper address space isolation	Full ELF64 parser validating magic/class/machine (EM_AARCH64). Per-process L2/L3 page table sub-trees with shared L0/L1. Non-Global (nG) bit on user PTEs. osai_process_aspace_t tracks private physical pages. Address space swap on context switch with TLBI VMALLE1IS.	✅ Done
3	Persistent filesystem across reboots	Separate osai-persistent.img VirtIO block device surviving reboots. MFS v3: 64 nodes, 16 blocks/file, 256 data sectors. fsck_volume() consistency check on mount. Handle-based multi-device virtio_blk API.	✅ Done
4	Real TCP/IP stack with external connectivity	Persistent VirtIO-net driver with live TX/RX queues. ARP cache, IPv4 headers, ICMP echo reply. Static IP 10.0.2.15 via QEMU SLIRP. Socket syscall ABI (LISTEN/ACCEPT/RECV/SEND/CLOSE). network_poll_tick() for scheduler-driven RX.	✅ Done
5	Real SSH server	Userspace /bin/sshd on port 22. Crypto from scratch: SHA-256 (FIPS 180-4), HMAC-SHA256 (RFC 2104), AES-128-CTR (FIPS 197), Curve25519 (RFC 7748). SSH-2 protocol: KEX, user auth, session channels. Baked-in test keypair, hardcoded admin:admin.	✅ Done

---

Tier 2: Operability

Without these, the system is not manageable in any real deployment.

#	Item	Description	Status
6	Real-time clock and wall time	PL031 RTC driver at 0x01010000 reading epoch seconds. wall_time_now_ns() combines RTC epoch with monotonic counter offset for sub-second wall-clock resolution. wall_time_calibrate() called once at boot. Self-test verifies monotonicity and epoch lower bound.	✅ Done
7	Proper logging daemon with persistent log storage	64 KB in-kernel ring buffer (klog_ring.c). Log levels (DEBUG/INFO/WARN/ERROR/PANIC) with klog_level() prepending wall-clock timestamps. klog_flush() persists ring to /var/log/kern.log via mutable_fs. klog_rotate() at 8 KB file limit. All klog() output captured to ring simultaneously with UART.	✅ Done
8	Process crash recovery and restart policies	Exponential backoff (1s base, doubling to 60s cap). "on-failure" restart policy (restart on crash, not clean exit). 8-slot crash dump ring recording service name, exit code, wall-clock timestamp, restart count, and uptime. service_heartbeat() updates per-service heartbeat time. service_watchdog_check() detects hung services (30s timeout).	✅ Done
9	Package/update delivery mechanism	Freestanding FIPS 180-4 SHA-256 implementation (sha256.c), verified with NIST test vectors. update_stage_chunk() for chunked payload delivery (4 KB max chunks) to /update/staging.bin. Running SHA-256 hash verification via update_verify_hash(). Key-value manifest parser (update_parse_manifest()) extracting version, target, size, hash, generation. Delivery status tracking (bytes received/expected, chunks written, hash verified).	✅ Done
10	Hardware watchdog and panic recovery	Timer-based watchdog with 10s deadline. watchdog_trigger_reset() issues PSCI SYSTEM_RESET via HVC #0 (0x84000009), replacing the infinite wfe halt in panic_at(). Boot counter persisted to /var/boot_counter via mutable_fs. Recovery mode activated after 3 consecutive failed boots (skips non-essential services, attempts update recovery). Boot counter reset on successful boot completion.	✅ Done

---

Tier 3: AI Agent Capability

Without these, OSAI can't do its actual job — running embedded AI agents that improve applications.

#	Item	Description	Status
11	Real CPU inference runtime with actual model execution	The CPU AI runtime is a deterministic skeleton with fixed outputs. Production needs real matrix/tensor operations, at minimum GGUF or similar model format loading, quantized INT8/INT4 kernels, attention computation, and tokenizer execution for small models (e.g., Phi-3-mini, Llama-3-8B quantized).	⬜ Not started
12	Real source-code indexing	The source index loads a 2-file fixture. Production needs to actually parse and index a real Git repository — walk the tree, tokenize source files, build symbol tables, and maintain dependency graphs.	⬜ Not started
13	Real Git operations	The Git workspace agent models metadata/lifecycle but doesn't speak Git protocol. Production needs at minimum libgit2-equivalent functionality: clone, diff, apply patch, commit, and push over SSH or HTTPS.	⬜ Not started
14	Real build/test sandbox execution	The sandbox models build/test as metadata transitions. Production needs to actually fork a process, run a compiler or build tool inside the sandbox with filesystem and network restrictions, capture stdout/stderr, and report pass/fail.	⬜ Not started
15	App-agent command protocol	No actual external API exists for sending requests to an AI cell. Production needs a defined protocol (HTTP/REST, gRPC, or custom TCP) that accepts agent requests, dispatches them to the correct AI cell, and streams responses back.	⬜ Not started

---

Tier 4: Hardware Readiness

Without these, OSAI won't survive real deployment on physical hardware.

#	Item	Description	Status
16	NUMA-aware memory allocation	NUMA topology module (numa.c) creates single node spanning all conventional memory on QEMU virt. Per-node free-stacks (262K pages/node max). pmm_alloc_page_on_node(node_id), pmm_alloc_page_near(preferred) for locality-aware allocation. pmm_node_of_page() for page-to-node lookup. Existing pmm_alloc_page() delegates to node 0 with multi-node fallback. Self-test verifies topology, alloc/free, node lookup.	✅ Done
17	IOMMU/SMMU enforcement	SMMUv3 driver (smmu.c) at MMIO 0x09050000 (page 0 + page 1). IDR0 detection with graceful bypass fallback. Linear stream table (32 STEs × 64B). 128-entry command queue with producer/consumer tracking. smmu_register_stream()/smmu_unregister_stream() for device stream management. smmu_tlb_invalidate_all() via TLBI command. CR0.SMMUEN enable with CR0ACK polling. GBPA bypass for unmapped streams. smmu=on added to QEMU machine options.	✅ Done
18	Kernel ASLR and stack canaries	Stack canary seeded from CNTVCT_EL0 XOR'd with 0x4F5341495F43414E in entry.S after BSS clear. OSAI_STACK_PROTECT_BEGIN/END macros for manual insertion: XOR canary with SP, push/pop/compare. stack_canary_check() panics on mismatch. Self-test verifies canary seed, protected function execution, and deliberate corruption detection. Applied to critical kernel paths.	✅ Done
19	Rate limiting and resource quotas per AI cell	AI cells declare resources upfront but there's no runtime enforcement of CPU time, memory bandwidth, or network bandwidth quotas during operation. A misbehaving cell could starve others.	⬜ Not started
20	Proper PCI/PCIe enumeration beyond discovery	The x86_64 path does basic PCI discovery but no proper bus enumeration, BAR sizing, capability walking, or MSI/MSI-X interrupt setup needed for real NVMe and high-performance NICs.	⬜ Not started

---

Tier 5: Ecosystem

Without these, the project won't be adopted by external developers.

#	Item	Description	Status
21	Selective POSIX compatibility layer	Not full POSIX, but enough to run common tools: a shell with pipes/redirection, cat, ls, cp, basic sed/grep. Developers need to debug and inspect the system interactively.	⬜ Not started
22	CI/CD pipeline with automated QEMU regression	The gate scripts exist but there's no GitHub Actions or CI configuration. Every push needs automated smoke boot, regression tests, and report generation.	⬜ Not started
23	Performance benchmarking against Linux/BSD baselines	No comparative data exists. Production credibility requires running the same workloads on tuned Linux (with isolcpus, CPU pinning, kernel bypass) and showing where OSAI actually wins.	⬜ Not started
24	Developer documentation for external contributors	No API docs, no architecture guide beyond the wiki, no "getting started" for someone who wants to write an app-agent or contribute a driver.	⬜ Not started
25	Licensing decision	Still "to be decided." No one can use or contribute to an unlicensed project in production.	⬜ Not started

---

Effort Estimates

Tier	Scope	Items	Rough estimate
Tier 1	Core OS functionality	1-5	6-12 months Complete (commit 8083940)
Tier 2	Operability	6-10	2-4 months Complete (2025-06-16)
Tier 3	AI agent capability	11-15	4-8 months
Tier 4	Hardware readiness	16-20	3-6 months
Tier 5	Ecosystem	21-25	Ongoing

Critical path: Items 1-10 are complete (Tier 1 + Tier 2). Next critical path: Items 11-15 (Tier 3 AI Agent Capability) — without real model execution, source indexing, Git operations, and sandbox execution, OSAI can't perform its core AI agent mission.

Dependencies

Tier 1 (items 1-5)
  └── Tier 2 (items 6-10)
        └── Tier 3 (items 11-15)
              └── Tier 4 (items 16-20)
                    └── Tier 5 (items 21-25)

Tier 1 items are largely independent of each other but all must be complete before Tier 2 makes sense. Tier 3 requires Tier 2 operability (logging, crash recovery, updates). Tier 4 requires Tier 3 to be working in QEMU before porting to physical hardware. Tier 5 is ongoing but becomes priority after Tier 4.

Related Pages

• Project Tracker
• Implementation Plan
• QEMU 100 Completion Plan
• Roadmap
• Open Decisions
• Risk Register