CPU AI Runtime

CPU AI Runtime

Purpose

This page describes the CPU-only model runtime assumptions for OSAI.

Contents

• CPU-Only Model Serving
• Shared Weights and Private State
• Model Loader
• Tokenizer Boundary
• Acceleration Boundary

CPU-Only Model Serving

OSAI targets CPU-bound AI models. The runtime should support quantized local models and predictable inference loops without CUDA, Metal, GPU, or NPU dependency.

The first runtime should be minimal and measurable. It does not need to support every model format before the OS proves boot, memory, core isolation, networking, and telemetry.

Shared Weights and Private State

Shared read-only model weights are central to OSAI. A host running many app agents should avoid loading duplicate copies of the same model.

Each agent gets:

• a shared model-weight mapping;
• private KV/cache arenas;
• private tokenizer/request state;
• private source-code context;
• explicit permissions.

Model Loader

The loader should:

• validate model metadata;
• map weights read-only;
• prefer hugepage/large-page mappings where available;
• prefault hot regions before READY;
• expose model identity for sharing;
• report memory placement and faults.

Tokenizer Boundary

Tokenization can run outside the hottest inference loop if that improves isolation. Tokenizer memory and source-context memory should be accounted separately from model weights and KV/cache.

Acceleration Boundary

Architecture-specific CPU acceleration is allowed through generic fallbacks:

• scalar fallback first;
• x86-64 vector paths where available;
• AArch64 vector paths where available;
• AMX or similar matrix extensions only behind feature detection.

GPU acceleration is not a requirement and must not be necessary for correctness.

Related Pages

• Memory System
• Performance Targets
• Intel Xeon Port