QEMU Full OS Core Workdown

QEMU Full OS Core Workdown

Purpose

This page tracks the work required to move OSAI from a QEMU readiness gate to a real QEMU full OS core.

The QEMU hardware-readiness gate is useful, but it is not the same as a complete OS. Intel hardware work should not start until the QEMU core has stable process, userspace control-plane, persistence, networking, CPU-AI runtime, security, telemetry, and filesystem contracts.

Contents

• Completion Rule
• Phase Q1: Real Process Lifecycle
• Phase Q2: Real Userspace Control Plane
• Phase Q3: Real Filesystem and Persistence
• Phase Q4: Networking Depth
• Phase Q5: CPU-Only AI Runtime Depth
• Phase Q6: Security Enforcement
• Phase Q7: QEMU Release Candidate
• Phase Q8: Full Filesystem
• Phase Q9: Process Supervisor
• Phase Q10: Network Stack Maturity
• Phase Q11: CPU-Only Model Runtime
• Phase Q12: AI Cell Production Contract

Completion Rule

Do not start Intel hardware code until this QEMU workdown has a release candidate gate with stable ABI, telemetry, filesystem, process, security, and control-plane contracts.

Phase Q1: Real Process Lifecycle

Goal: stop treating /init as a one-way kernel-hosted test and make EL0 process execution return cleanly to kernel supervision.

Current status:

• Done: process states: empty, loaded, running, exited, failed.
• Done: process lifecycle transition counters.
• Done: sys_exit marks the current process exited or failed.
• Done: EL0 exit returns to a kernel continuation.
• Done: process lifecycle telemetry.
• Done: child service descriptor parsing from /etc/osai-init.conf.
• Done: /init defines, starts, and reports the child /svc/source-index service record through osctl.
• Done: invalid-transition and failed-exit lifecycle tests.

Definition of done: QEMU boot proves /init loads, runs, exits, and the kernel resumes after EL0. Telemetry includes loaded, running, exited, failed, transition counters, child service descriptor count, and service transition count.

Verification:

• make qemu-smoke
• make qemu-readiness-gate

Phase Q2: Real Userspace Control Plane

Goal: move policy decisions out of kernel fixtures and into userland service management.

Required work:

• Done: split userland into /init and /bin/service-manager payloads.
• Done: add /etc/services/source-index.svc as the first read-only service descriptor.
• Done: add descriptor-read and service status/start/stop/restart/rollback/update syscalls.
• Done: keep service operations behind capability checks.
• Done: prove missing rollback/update capabilities fail closed.
• Done: emit control-plane syscall, denial, and descriptor-read telemetry.

Definition of done: QEMU boot proves /bin/service-manager runs as a second EL0 process, reads the ROFS service descriptor, manages /svc/source-index, and reports control-plane telemetry.

Verification:

• make qemu-smoke
• make qemu-readiness-gate

Phase Q3: Real Filesystem and Persistence

Goal: replace metadata-only persistence with serialized state records and mutable VirtIO-backed state.

Required work:

• Done: mutable state region on the VirtIO test block image.
• Done: serialized boot, service, workspace, sandbox, and update records.
• Done: checksum and version validation for persistence records.
• Done: persisted records reload after a QEMU reboot.
• Done: rollback restores prior service and workspace records.
• Done: read-only system image and mutable state boundaries are explicit.

Definition of done: make qemu-persistence-reboot proves the same VirtIO state image survives a two-boot QEMU run, reports persistence_boot_loads, and validates rollback after reload.

Verification:

• make qemu-smoke
• make qemu-persistence-reboot
• make qemu-readiness-gate

Phase Q4: Networking Depth

Goal: move beyond parser smoke paths into queue-backed packet flow.

Required work:

• Done: queue-backed RX/TX packet ownership.
• Done: UDP flow table with queue/core ownership.
• Done: minimal TCP state machine with SYN timeout accounting.
• Done: packet lifetime/drop counters.
• Done: malformed packet tests remain enforced.
• Done: p50/p95/p99/p999 correctness telemetry.

Definition of done: make qemu-smoke proves packets move through queue metadata, creates a UDP flow, expires a TCP SYN flow, and reports packet lifecycle/drop counters plus p999 latency telemetry.

Verification:

• make qemu-smoke
• make qemu-readiness-gate

Phase Q5: CPU-Only AI Runtime Depth

Goal: turn the deterministic CPU-AI stub into real boundaries that can later host actual CPU-only model code.

Required work:

• Done: model-loader interface and manifest format.
• Done: tokenizer/runtime boundary.
• Done: shared read-only model weights.
• Done: private KV/cache arenas per AI Cell.
• Done: multi-cell shared-weight tests.
• Done: model load failure tests.
• Done: GPU acceleration remains outside the runtime contract.

Definition of done: make qemu-smoke requires model load, load failure, tokenizer, runtime, KV/cache write, shared-weight bind, and GPU rejection telemetry.

Verification:

• make qemu-smoke
• make qemu-readiness-gate

Phase Q6: Security Enforcement

Goal: make capability policy enforce service, filesystem, Git/workspace, update, rollback, and build/test permissions.

Required work:

• Complete: extend capability checks across service operations.
• Complete: enforce filesystem read/write boundaries.
• Complete: enforce Git workspace write and patch permissions.
• Complete: enforce build/test sandbox permissions.
• Complete: strengthen signed update validation beyond the current stub.
• Complete: add rollback authorization tests.
• Complete: ensure secrets are rejected from logs, updates, and benchmark records.

Definition of done: denied operations fail closed and emit telemetry counters for capability, filesystem, workspace, sandbox, rollback, update-policy, credential, and signature rejection paths.

Verification:

• make qemu-smoke
• make qemu-readiness-gate

Phase Q7: QEMU Release Candidate

Goal: freeze the QEMU full OS core contract.

Required work:

• Complete: expand make qemu-readiness-gate to include CPU matrix tiers.
• Complete: freeze syscall ABI.
• Complete: freeze telemetry schema.
• Complete: freeze filesystem format.
• Complete: freeze persistence record format.
• Complete: freeze service descriptor format.
• Complete: document what remains out of scope for Intel bring-up.

Definition of done: a single QEMU release-candidate gate proves process, userspace control, persistence, networking, CPU-AI runtime, security, CPU matrix, and documentation contracts are stable enough to start Intel Desktop work.

Release-candidate contract:

• contracts/qemu-rc-v1.json
• schema: osai.qemu.release_candidate_contract.v1
• generated CPU report: build/qemu-cpu-matrix-report.json

Phase Q8: Full Filesystem

Goal: move beyond the read-only boot filesystem and single-sector persistence records toward a real mutable filesystem area.

Required work:

• Complete: reserve a separate VirtIO-backed mutable filesystem sector range.
• Complete: add checksum-protected filesystem metadata.
• Complete: add explicit read-write mount policy.
• Complete: add create/update/read/delete file operations.
• Complete: add commit and rollback boundaries for mutable files.
• Complete: add QEMU smoke and benchmark gates for mutable filesystem telemetry.
• Complete: add a real block allocator instead of fixed one-sector file slots.
• Complete: add directory records and path traversal beyond fixed prefixes.
• Complete: add larger files spanning multiple sectors.
• Complete: add crash-consistency transaction replay.
• Complete: connect service, workspace, and update state to the mutable filesystem.

Definition of done: QEMU proves mutable filesystem mount, mutation, checksum validation, allocation, directory records, multi-sector files, journal replay, delete behavior, reboot load, and rollback behavior without weakening the read-only boot filesystem boundary. A later production-ready filesystem can add a POSIX-like surface and larger storage area, but the QEMU contract no longer depends on fixed one-sector file slots.

Phase Q9: Process Supervisor

Goal: make the service manager behave like a small supervisor instead of a single child-service smoke test.

Required work:

• Complete: track parent-child service tree edges.
• Complete: store child parent metadata in the service record.
• Complete: allow service configuration to apply to child services, not only /init.
• Complete: enforce child restart policy and max_restarts.
• Complete: add a controlled child crash path through the userspace service manager.
• Complete: restart a failed child when policy allows it.
• Complete: clean up service runtime state after exit or crash.
• Complete: count service logs globally and per service.
• Complete: count process address-space reclamation after EL0 process exit.
• Complete: add smoke, benchmark, readiness, and contract gates for supervisor telemetry.

Definition of done: /bin/service-manager defines /svc/source-index as a child of /init, starts it, proves restart denial while policy is never, reconfigures it to always, logs a service event, injects a controlled crash, and observes the supervisor cleanup/restart path. Telemetry includes service_tree_edges, service_restarts, service_crashes, service_cleanups, service_log_records, and user_process_reclaims.

Phase Q10: Network Stack Maturity

Goal: turn the QEMU network path from a queue-backed smoke fixture into a stricter TCP/UDP lifecycle contract.

Required work:

• Complete: route packets through deterministic flow-to-queue selection.
• Complete: keep established UDP flows pinned to their owning queue and cell.
• Complete: add explicit RX/TX/completion accounting for queue rings.
• Complete: add UDP flow hit tracking.
• Complete: add UDP idle expiry.
• Complete: add TCP established/closed counters.
• Complete: add TCP retransmit-before-timeout policy for SYN-received flows.
• Complete: keep TCP timeout behavior after retransmit.
• Complete: add telemetry for queue backpressure drops and flow/core mismatches.
• Complete: expand smoke, benchmark, readiness, and RC contract gates.

Definition of done: QEMU boot proves UDP flow reuse, UDP expiry, TCP establishment, TCP retransmit, TCP timeout, queue RX/TX/completion accounting, zero queue backpressure drops, and zero flow/core mismatches. make qemu-smoke, python3 ./scripts/qemu-benchmark.py, and make qemu-readiness-gate require the mature networking telemetry.

Phase Q11: CPU-Only Model Runtime

Goal: turn the earlier CPU-AI runtime boundary into a file-backed QEMU model runtime contract.

Required work:

• Complete: add /models/cpu-ai-mvp.osaimodel to the VirtIO-backed read-only filesystem.
• Complete: increase the ROFS header to 1024 bytes so the filesystem can carry the model file cleanly.
• Complete: validate model magic, version, quantization, flags, ranges, tokenizer size, private KV/cache requirement, and payload checksum.
• Complete: reject missing model files, checksum failures, undersized tokenizer tables, GPU-required models, and undersized KV/cache bindings.
• Complete: bind a tokenizer table from the model file instead of hardcoded token ids.
• Complete: dispatch deterministic CPU decode through a runtime id and count kernel dispatches.
• Complete: keep shared model weights read-only and private KV/cache per AI Cell.
• Complete: expand smoke, benchmark, readiness, telemetry, and RC contract gates.

Definition of done: QEMU boot proves a CPU-only model file is loaded from ROFS, admitted into a shared read-only model arena, used by multiple AI Cells with private KV/cache arenas, and rejected correctly for malformed or GPU-required model configurations. Telemetry includes model file loads/rejects, loaded bytes, manifest validations, tokenizer binds, CPU kernel dispatches, admission rejects, and checksum failures.

Phase Q12: AI Cell Production Contract

Goal: turn AI Cell setup from fixture-style manifests into a stable resource contract that later platform ports can preserve.

Required work:

• Complete: define the AIC1 version 1 descriptor ABI with fixed 112-byte layout.
• Complete: require CPU-only AI, fixed core, shared-model, private KV/cache, NIC queue, and Git workspace flags.
• Complete: validate descriptor magic, version, size, reserved fields, bounded names, and checksum.
• Complete: admit descriptors through the existing AI Cell manifest validation path.
• Complete: account PMM/VMM-backed arena reservations in current and peak page/byte counters.
• Complete: bind AI Cells to real network queue ownership and release those queues during destroy.
• Complete: track workspace bind and release lifecycle.
• Complete: reject duplicate core leases, missing model arenas, duplicate NIC queues, and duplicate workspaces.
• Complete: keep shared model weights reusable while private KV/cache remains per AI Cell.
• Complete: expand smoke, benchmark, readiness, telemetry, and RC contract gates for descriptor and resource-contract counters.

Definition of done: QEMU boot proves valid AI Cell descriptors are accepted, malformed descriptors are rejected, resource conflicts fail closed, arenas are reserved and released, queue/workspace ownership is balanced, and the final telemetry reports zero currently reserved AI Cell arena pages with nonzero peak usage. make qemu-smoke, python3 ./scripts/qemu-benchmark.py, and make qemu-readiness-gate require the AI Cell contract telemetry.

Related Pages

• QEMU on macOS
• QEMU CPU Test Matrix
• Hardware Readiness Gate
• Filesystem and Storage
• Implementation Plan
• Roadmap