Production Readiness Roadmap

Production Readiness Roadmap

Purpose

This page defines what OSAI needs to become useful for production deployment. The QEMU target is complete (milestones 0-71), but the OS is still a single-boot, cooperative, self-test-oriented prototype. This roadmap covers the gap between "QEMU complete" and "production useful."

Items are ordered by priority within each tier. Higher tiers block lower tiers.

Tier 1 is complete as of commit 8083940 (2025-06-16). All 5 core OS items implemented: preemptive scheduler, ELF loader with address space isolation, persistent filesystem, TCP/IP stack, and SSH server. 53 files changed, ~3,900 lines added.

Tier 2 is complete as of 2025-06-16. All 5 operability items implemented: RTC/wall time, persistent logging with ring buffer, crash recovery with backoff, package delivery with SHA-256 verification, and hardware watchdog with panic recovery. 8 new files, 10 modified files.

Status Legend

Marker	Meaning
⬜ Not started	Planned but no implementation yet.
🔵 In progress	Active engineering work.
✅ Done	Implemented and tested.
⚠️ Blocked	Waiting on prerequisite design, code, hardware, or tooling.

Contents

• Tier 1: Core OS Functionality
• Tier 2: Operability
• Tier 3: AI Agent Capability
• Tier 4: Hardware Readiness
• Tier 5: Ecosystem
• Effort Estimates

---

Tier 1: Core OS Functionality

Without these, nothing real can run. These are the critical-path items.

#	Item	Description	Status
1	Real preemptive scheduler with concurrent execution	Round-robin scheduler on CPU 0 with GICv3 PPI timer interrupts (INTID 30). Full context save/restore (34 registers). CPUs 1-3 remain leased AI cores with no-migration guarantee. scheduler_tick() drives preemption and address space switching.	✅ Done
2	Real ELF loader with proper address space isolation	Full ELF64 parser validating magic/class/machine (EM_AARCH64). Per-process L2/L3 page table sub-trees with shared L0/L1. Non-Global (nG) bit on user PTEs. osai_process_aspace_t tracks private physical pages. Address space swap on context switch with TLBI VMALLE1IS.	✅ Done
3	Persistent filesystem across reboots	Separate osai-persistent.img VirtIO block device surviving reboots. MFS v3: 64 nodes, 16 blocks/file, 256 data sectors. fsck_volume() consistency check on mount. Handle-based multi-device virtio_blk API.	✅ Done
4	Real TCP/IP stack with external connectivity	Persistent VirtIO-net driver with live TX/RX queues. ARP cache, IPv4 headers, ICMP echo reply. Static IP 10.0.2.15 via QEMU SLIRP. Socket syscall ABI (LISTEN/ACCEPT/RECV/SEND/CLOSE). network_poll_tick() for scheduler-driven RX.	✅ Done
5	Real SSH server	Userspace /bin/sshd on port 22. Crypto from scratch: SHA-256 (FIPS 180-4), HMAC-SHA256 (RFC 2104), AES-128-CTR (FIPS 197), Curve25519 (RFC 7748). SSH-2 protocol: KEX, user auth, session channels. Baked-in test keypair, hardcoded admin:admin.	✅ Done

---

Tier 2: Operability

Without these, the system is not manageable in any real deployment.

#	Item	Description	Status
6	Real-time clock and wall time	PL031 RTC driver at 0x01010000 reading epoch seconds. wall_time_now_ns() combines RTC epoch with monotonic counter offset for sub-second wall-clock resolution. wall_time_calibrate() called once at boot. Self-test verifies monotonicity and epoch lower bound.	✅ Done
7	Proper logging daemon with persistent log storage	64 KB in-kernel ring buffer (klog_ring.c). Log levels (DEBUG/INFO/WARN/ERROR/PANIC) with klog_level() prepending wall-clock timestamps. klog_flush() persists ring to /var/log/kern.log via mutable_fs. klog_rotate() at 8 KB file limit. All klog() output captured to ring simultaneously with UART.	✅ Done
8	Process crash recovery and restart policies	Exponential backoff (1s base, doubling to 60s cap). "on-failure" restart policy (restart on crash, not clean exit). 8-slot crash dump ring recording service name, exit code, wall-clock timestamp, restart count, and uptime. service_heartbeat() updates per-service heartbeat time. service_watchdog_check() detects hung services (30s timeout).	✅ Done
9	Package/update delivery mechanism	Freestanding FIPS 180-4 SHA-256 implementation (sha256.c), verified with NIST test vectors. update_stage_chunk() for chunked payload delivery (4 KB max chunks) to /update/staging.bin. Running SHA-256 hash verification via update_verify_hash(). Key-value manifest parser (update_parse_manifest()) extracting version, target, size, hash, generation. Delivery status tracking (bytes received/expected, chunks written, hash verified).	✅ Done
10	Hardware watchdog and panic recovery	Timer-based watchdog with 10s deadline. watchdog_trigger_reset() issues PSCI SYSTEM_RESET via HVC #0 (0x84000009), replacing the infinite wfe halt in panic_at(). Boot counter persisted to /var/boot_counter via mutable_fs. Recovery mode activated after 3 consecutive failed boots (skips non-essential services, attempts update recovery). Boot counter reset on successful boot completion.	✅ Done

---

Tier 3: AI Agent Capability

Without these, OSAI can't do its actual job — running embedded AI agents that improve applications.

#	Item	Description	Status
11	Real CPU inference runtime with actual model execution	The CPU AI runtime is a deterministic skeleton with fixed outputs. Production needs real matrix/tensor operations, at minimum GGUF or similar model format loading, quantized INT8/INT4 kernels, attention computation, and tokenizer execution for small models (e.g., Phi-3-mini, Llama-3-8B quantized).	⬜ Not started
12	Real source-code indexing	The source index loads a 2-file fixture. Production needs to actually parse and index a real Git repository — walk the tree, tokenize source files, build symbol tables, and maintain dependency graphs.	⬜ Not started
13	Real Git operations	The Git workspace agent models metadata/lifecycle but doesn't speak Git protocol. Production needs at minimum libgit2-equivalent functionality: clone, diff, apply patch, commit, and push over SSH or HTTPS.	⬜ Not started
14	Real build/test sandbox execution	The sandbox models build/test as metadata transitions. Production needs to actually fork a process, run a compiler or build tool inside the sandbox with filesystem and network restrictions, capture stdout/stderr, and report pass/fail.	⬜ Not started
15	App-agent command protocol	No actual external API exists for sending requests to an AI cell. Production needs a defined protocol (HTTP/REST, gRPC, or custom TCP) that accepts agent requests, dispatches them to the correct AI cell, and streams responses back.	⬜ Not started

---

Tier 4: Hardware Readiness

Without these, OSAI won't survive real deployment on physical hardware.

#	Item	Description	Status
16	NUMA-aware memory allocation	NUMA topology module (numa.c) creates single node spanning all conventional memory on QEMU virt. Per-node free-stacks (262K pages/node max). pmm_alloc_page_on_node(node_id), pmm_alloc_page_near(preferred) for locality-aware allocation. pmm_node_of_page() for page-to-node lookup. Existing pmm_alloc_page() delegates to node 0 with multi-node fallback. Self-test verifies topology, alloc/free, node lookup.	✅ Done
17	IOMMU/SMMU enforcement	SMMUv3 driver (smmu.c) at MMIO 0x09050000 (page 0 + page 1). IDR0 detection with graceful bypass fallback. Linear stream table (32 STEs × 64B). 128-entry command queue with producer/consumer tracking. smmu_register_stream()/smmu_unregister_stream() for device stream management. smmu_tlb_invalidate_all() via TLBI command. CR0.SMMUEN enable with CR0ACK polling. GBPA bypass for unmapped streams. smmu=on added to QEMU machine options.	✅ Done
18	Kernel ASLR and stack canaries	Stack canary seeded from CNTVCT_EL0 XOR'd with 0x4F5341495F43414E in entry.S after BSS clear. OSAI_STACK_PROTECT_BEGIN/END macros for manual insertion: XOR canary with SP, push/pop/compare. stack_canary_check() panics on mismatch. Self-test verifies canary seed, protected function execution, and deliberate corruption detection. Applied to critical kernel paths.	✅ Done
19	Rate limiting and resource quotas per AI cell	Token bucket rate limiter (rate_limit.c) with per-cell CPU, network RX/TX, and memory quotas. rate_limit_cpu_consume() throttles on exhaustion. rate_limit_check_memory()/rate_limit_memory_commit()/rate_limit_memory_release() for memory cap enforcement. rate_limit_check_network() drops frames exceeding bandwidth. Lazy refill via timer_now_ns(). Violation counters per resource type. Self-test exercises exhaustion, cap violations, and release.	✅ Done
20	Proper PCI/PCIe enumeration beyond discovery	The x86_64 path does basic PCI discovery but no proper bus enumeration, BAR sizing, capability walking, or MSI/MSI-X interrupt setup needed for real NVMe and high-performance NICs.	⬜ Not started

---

Tier 5: Ecosystem

Without these, the project won't be adopted by external developers.

#	Item	Description	Status
21	Selective POSIX compatibility layer	Not full POSIX, but enough to run common tools: a shell with pipes/redirection, cat, ls, cp, basic sed/grep. Developers need to debug and inspect the system interactively.	⬜ Not started
22	CI/CD pipeline with automated QEMU regression	The gate scripts exist but there's no GitHub Actions or CI configuration. Every push needs automated smoke boot, regression tests, and report generation.	⬜ Not started
23	Performance benchmarking against Linux/BSD baselines	No comparative data exists. Production credibility requires running the same workloads on tuned Linux (with isolcpus, CPU pinning, kernel bypass) and showing where OSAI actually wins.	⬜ Not started
24	Developer documentation for external contributors	No API docs, no architecture guide beyond the wiki, no "getting started" for someone who wants to write an app-agent or contribute a driver.	⬜ Not started
25	Licensing decision	Still "to be decided." No one can use or contribute to an unlicensed project in production.	⬜ Not started

---

Effort Estimates

Tier	Scope	Items	Rough estimate
Tier 1	Core OS functionality	1-5	6-12 months Complete (commit 8083940)
Tier 2	Operability	6-10	2-4 months Complete (2025-06-16)
Tier 3	AI agent capability	11-15	4-8 months
Tier 4	Hardware readiness	16-20	3-6 months
Tier 5	Ecosystem	21-25	Ongoing

Critical path: Items 1-10 are complete (Tier 1 + Tier 2). Next critical path: Items 11-15 (Tier 3 AI Agent Capability) — without real model execution, source indexing, Git operations, and sandbox execution, OSAI can't perform its core AI agent mission.

Dependencies

Tier 1 (items 1-5)
  └── Tier 2 (items 6-10)
        └── Tier 3 (items 11-15)
              └── Tier 4 (items 16-20)
                    └── Tier 5 (items 21-25)

Tier 1 items are largely independent of each other but all must be complete before Tier 2 makes sense. Tier 3 requires Tier 2 operability (logging, crash recovery, updates). Tier 4 requires Tier 3 to be working in QEMU before porting to physical hardware. Tier 5 is ongoing but becomes priority after Tier 4.

Related Pages

• Project Tracker
• Implementation Plan
• QEMU 100 Completion Plan
• Roadmap
• Open Decisions
• Risk Register