CPU AI Runtime

CPU AI Runtime

Purpose

This page describes the CPU-only model runtime assumptions for OSAI.

Contents

• CPU-Only Model Serving
• Shared Weights and Private State
• Model Loader
• Tokenizer Boundary
• Acceleration Boundary

CPU-Only Model Serving

OSAI targets CPU-bound AI models. The runtime should support quantized local models and predictable inference loops without CUDA, Metal, GPU, or NPU dependency.

The first runtime should be minimal and measurable. It does not need to support every model format before the OS proves boot, memory, core isolation, networking, and telemetry.

Shared Weights and Private State

Shared read-only model weights are central to OSAI. A host running many app agents should avoid loading duplicate copies of the same model.

Each agent gets:

• a shared model-weight mapping;
• private KV/cache arenas;
• private tokenizer/request state;
• private source-code context;
• explicit permissions.

Model Loader

The loader should:

• validate model metadata;
• map weights read-only;
• prefer hugepage/large-page mappings where available;
• prefault hot regions before READY;
• expose model identity for sharing;
• report memory placement and faults.

Initial model format support is an open implementation choice. GGUF is a practical first candidate for quantized CPU-only local models. SafeTensors may be useful for tensor metadata and validation later, but no format should be treated as final until recorded in Open Decisions.

Model Arena Layout

model identity metadata
  -> read-only weight mapping
  -> optional hugepage alignment padding
  -> per-platform CPU dispatch metadata

per-agent private state
  -> tokenizer/request state
  -> KV/cache arena
  -> source-context references

The model arena is shared read-only. The KV/cache arena is private per embedded app agent. This avoids duplicate model copies while preserving per-agent inference state.

Tokenizer Boundary

Tokenization can run outside the hottest inference loop if that improves isolation. Tokenizer memory and source-context memory should be accounted separately from model weights and KV/cache.

Acceleration Boundary

Architecture-specific CPU acceleration is allowed through generic fallbacks:

• scalar fallback first;
• x86-64 vector paths where available;
• AArch64 vector paths where available;
• AMX or similar matrix extensions only behind feature detection.

GPU acceleration is not a requirement and must not be necessary for correctness.

Implementation Requirements

• Model identity must include format, quantization, checksum, and architecture compatibility.
• Shared model weights must be mapped read-only in every AI Cell.
• Private KV/cache arenas must never alias another agent's private state.
• CPU feature dispatch must fall back safely when AMX, AVX-family features, or AArch64 vector features are unavailable.
• Tokenizer errors must fail the request without corrupting model arena state.

Related Pages

• Memory System
• Performance Targets
• Intel Xeon Port
• Benchmark Result Format