ALLM

Adaptive Local LLM runtime: CPU-first inference in Rust that runs models larger than RAM by streaming weights from storage under a hard memory budget — the same layerwise-streaming idea as AirLLM, built for CPU/RAM instead of GPU/VRAM.

ALLM streams transformer blocks through a byte-budgeted LRU cache and hides I/O latency behind computation using a background prefetch thread. A controller measures the device (RAM, CPU, storage speed, thermal and battery state) and chooses how much to cache and how many threads to use, all under a hard memory budget so the process is not OOM-killed on constrained devices such as phones running Termux.

$ allm run models/qwen.gguf -p "The capital of France is" --max-tokens 20
The capital of France is Paris. It is the largest city in France and the capital

Contents

• Why ALLM exists
• Architecture
• How streaming works
• Honest performance expectations
• Comparison with llama.cpp
• Supported platforms
• Installation
• Quickstart
• Commands
• Bindings
• Testing
• Android and Termux
• Project layout

Why ALLM exists

Running a large model on a small device usually fails for one reason: the weights do not fit in RAM. The common fallbacks are to pick a smaller model or to let the operating system swap, which is slow and can crash the process.

ALLM applies the layerwise streaming insight first demonstrated by AirLLM for GPU/VRAM: a transformer block is used for milliseconds, then not needed until the next token. Stream it in, compute, stream the next one. The difference: AirLLM targets GPU with Python/PyTorch. ALLM targets CPU/RAM with Rust, adds an LRU cache (keep K recently-used blocks, not just 1), enforces a hard budget ceiling, and adds a background prefetch thread to hide I/O latency — all with no GPU dependency.

Architecture

ALLM is a Cargo workspace of small crates with a one-way dependency direction. Each crate owns one concern and exposes a trait wherever a component is meant to be swappable.

graph TD
    core["allm-core<br/>types, errors, logging"]
    gguf["allm-gguf<br/>GGUF parser"]
    tok["allm-tokenizer<br/>BPE / tokenizer.json"]
    cache["allm-cache<br/>ByteBudgetCache (LRU + hard budget)"]
    stream["allm-streaming<br/>StorageBackend, prefetch"]
    adapt["allm-adaptive<br/>DeviceProfile, policy, controller"]
    compute["allm-compute<br/>ComputeBackend, Model, StreamingModel"]
    sched["allm-scheduler<br/>paged KV, continuous batching"]
    cli["allm (CLI)"]
    capi["allm-capi<br/>C ABI"]
    py["allm-python<br/>PyO3"]

    core --> gguf & tok & cache & stream & adapt & compute & sched
    gguf --> tok & compute
    cache --> compute
    compute --> cli & capi & py & sched
    stream --> cli
    adapt --> cli

    classDef front fill:#e8f0fe,stroke:#4285f4;
    class cli,capi,py front;

Loading

The three front ends (CLI, C API, Python) share one engine. Storage, cache, and adaptive logic do not depend on allm-compute, so the model code can change without disturbing them.

Key seams (traits):

Trait	Crate	Purpose
ComputeBackend	allm-compute	Device + matmul primitive. CandleBackend (CPU) is production; ScalarBackend proves the trait swaps and marks where a NEON/AVX kernel goes.
StorageBackend	allm-streaming	Byte source by offset: MmapStorage, DirectReadStorage, CompressedChunkStorage, CallbackStorage.
PolicyStrategy	allm-adaptive	Maps a DeviceProfile to a StreamingPolicy.
TokenSource	allm-scheduler	Lets the scheduler be driven by a mock or the real model.

See docs/ARCHITECTURE.md for the full design.

How streaming works

For each token, the model walks every transformer block. A resident model holds all blocks in RAM; a streaming model loads each block through the budgeted cache, reading and decoding it from the GGUF file on a miss and evicting the least-recently-used block when the budget would be exceeded. A background prefetch thread starts loading block i+1 while block i is computing, hiding disk I/O behind compute (the same technique AirLLM uses on the GPU).

flowchart LR
    P["prompt"] --> E["tokenizer.encode"]
    E --> EMB["embedding lookup<br/>(resident)"]
    EMB --> L{"for each block"}
    L -->|hit| ATTN["attention (RoPE, GQA, KV cache)<br/>+ SwiGLU MLP"]
    L -->|prefetch ready| ATTN
    L -->|miss| LOAD["load block from GGUF<br/>via ByteBudgetCache"]
    LOAD --> EV["evict LRU if over budget"]
    EV --> ATTN
    ATTN -->|"kick off prefetch(i+1)"| PF["prefetch thread<br/>(background)"]
    ATTN --> L
    L -->|done| N["final norm + lm_head"]
    N --> S["sampler<br/>(greedy / temp+top-k/top-p)"]
    S --> D["tokenizer.decode -> stream out"]
    S -->|next token| EMB

Loading

The hard budget is enforced by ByteBudgetCache: resident weight bytes never exceed the configured limit. This is verified by a test that runs the streaming model under a deliberately small budget and asserts its output is identical, token for token, to the resident model (see Testing).

Honest performance expectations

Streaming weights from storage is not free:

• Memory mapping plus the OS page cache already lets you run a model roughly twice the size of RAM. ALLM adds value by enforcing a hard budget so the device stays stable, and by hiding I/O latency through prefetching.
• When a model genuinely exceeds RAM, throughput drops. Reading weights from disk every token is slow. Large contiguous reads and prefetch help, but a 70B model on a phone will not feel like a 1B model.
• Large sequential reads dominate throughput; small random reads are latency bound. ALLM reads in large chunks.
• The key-value cache grows with context length and must be budgeted. On tight devices a context that is too large will get the process OOM-killed.

Measured on an 8-core x86-64, Windows, release build:

Qwen2.5-0.5B-Instruct (Q4 mix, 463 MiB, 24 blocks)

Mode	Budget	Decode	Cache stats
Resident	—	8.5 tok/s	—
Streaming (all blocks fit)	256 MiB	8.0 tok/s	498 hits, 6 misses, 0 evictions
Streaming (tight, ~6 blocks fit)	64 MiB	3.0 tok/s ↑ from 2.2	498 hits, 6 misses, 498 evictions

The 64 MiB result went from 2.2 → 3.0 tok/s (+36%) after wiring in background prefetch. All 498 "hits" are blocks committed by the prefetch thread via preload(); the 6 misses are the first few blocks before the prefetcher gets ahead.

Qwen2.5-3B-Instruct (Q4_K/Q6_K, 1.79 GiB, 36 blocks)

Mode	Budget	Decode	Cache stats
Resident	—	3.7 tok/s	—
Streaming (~5 blocks fit)	256 MiB	0.5 tok/s	751 hits, 5 misses, 751 evictions

The 3B model sees no speed gain from prefetch because each block's 43 MiB compute time (large matrix multiply) already exceeds its disk-load time — I/O completes "for free" inside the compute window. Prefetch still works correctly (751 hits vs 0 hits before), and the budget is respected in all cases (peak resident ≤ budget).

Comparison with llama.cpp and AirLLM

llama.cpp is the mature reference for local GGUF inference; ALLM is a young, CPU-first research runtime. AirLLM is the closest conceptual relative: it applies the same layerwise streaming idea but for GPU/VRAM. Measured on the same machine (Qwen2.5-0.5B Q4 mix):

Runtime	Prompt (pp32)	Decode
llama.cpp (b9849, 12 threads)	~222 tok/s	~32–36 tok/s
ALLM 0.1.0 resident (12 threads)	see note	8.5 tok/s
ALLM 0.1.0 streaming 256 MiB	see note	8.5 tok/s (all blocks cached)
ALLM 0.1.0 streaming 64 MiB	see note	2.2 tok/s (heavy eviction)

llama.cpp is 4–10× faster on decode, supports GPUs, far more architectures and tokenizers, and ships a server. AirLLM is Python/PyTorch and supports any HuggingFace model but is GPU-focused with no hard budget.

ALLM's distinguishing features: hard, device-adaptive RAM ceiling verified at every layer of every token + background prefetch (I/O overlaps compute, same as AirLLM) + LRU cache (K blocks, not just 1) + embeddable Rust crates

• Android/Termux target. Use llama.cpp for speed and breadth; AirLLM for any HF model on a GPU; ALLM when you specifically need a guaranteed budget on a CPU-only constrained device.

See docs/COMPARISON.md for the full feature matrix and AirLLM deep-dive, and docs/MEMORY_SEALING.md for the memory budget contract.

Supported platforms

• Operating systems: Windows, Linux, macOS, Android (Termux), Raspberry Pi.
• Architectures: x86-64, ARM64, ARMv7.

The first working target is a small GGUF model on Windows x86-64, then Linux, then Termux/Android ARM64 and Raspberry Pi.

Installation

Prerequisites

• Rust 1.82 or newer (rustup recommended): https://rustup.rs

• A C compiler, needed by the tokenizer's onig regex dependency:

  • Linux: gcc/clang (for example build-essential)
  • macOS: the Xcode command line tools (xcode-select --install)
  • Windows: the MSVC build tools (the "Desktop development with C++" workload)

  If you only ever load tokenizers from a tokenizer.json, the byte-level BPE path does not need onig at run time; see docs/ANDROID.md for building without it.

Option 1: install the allm binary from source

This builds in release mode and places allm on your PATH (in ~/.cargo/bin):

git clone https://github.com/allm-runtime/allm
cd allm
cargo install --path cli
allm about     # ALLM (Adaptive Local LLM runtime) version 0.1.0

To update later, re-run cargo install --path cli (add --force to overwrite), and to remove it: cargo uninstall allm-cli.

Option 2: build without installing

cargo build --release
./target/release/allm about

Option 3: prebuilt binaries

Tagged releases publish prebuilt archives for Linux, Windows, and macOS on the Releases page (built by .github/workflows/release.yml). Download the archive for your platform, extract it, and put the allm binary on your PATH.

Get a model

ALLM runs GGUF files. Download a small one to start, for example a Qwen2.5-0.5B-Instruct GGUF (about 460 MiB) from Hugging Face, then point the commands below at it. Qwen works out of the box; for models that ship a SentencePiece tokenizer, also pass --tokenizer-json <path>.

Quickstart

The toolchain is standard Rust (1.82 or newer).

cargo build --release
cargo test
cargo run --release -p allm-cli -- about

Running a model

ALLM runs GGUF models. Qwen2.5-0.5B-Instruct works out of the box because ALLM reconstructs its byte-level BPE tokenizer from the GGUF:

# download a small GGUF (about 460 MiB), then:
allm inspect models/qwen.gguf
allm run models/qwen.gguf -p "The capital of France is" --max-tokens 30

Sampling is greedy by default. Add --temperature 0.8 --top-k 40 --top-p 0.95 --seed 1 for sampled (reproducible) generation. Models that use a SentencePiece tokenizer (many llama models) are not reconstructed from GGUF yet; pass one with --tokenizer-json <path>.

Streaming under a memory budget

allm run models/qwen.gguf -p "Hello" --mem-budget 64MB

Blocks are loaded on demand through an LRU cache that never exceeds the budget, and the run prints cache statistics (resident peak, hits, misses, evictions). To let ALLM size the budget to your device automatically:

allm run models/qwen.gguf -p "Hello" --explain-policy

This probes RAM, CPU, and storage speed and prints the chosen budget, prefetch depth, and thread count with the reasoning. See docs/ADAPTIVE.md.

Commands

Command	What it does
allm about	Print the runtime name and version.
allm inspect <model.gguf> [--full]	Architecture, metadata, and tensor directory. --full prints every metadata entry.
allm tokenize "<text>" --model <m.gguf>	Encode and decode text. --tokenizer-json <path> loads a tokenizer.json instead.
allm run <model.gguf> -p "<prompt>"	Generate text, streaming tokens to stdout. See flags below.
allm doctor [--quick]	Print the device profile. --quick skips the storage micro-benchmark.
allm bench-storage <file> [--total 256MB]	Read-throughput across chunk sizes for the mmap and direct backends.
allm bench-backend [--size 512] [--backend both] [--iters 5]	Median GFLOP/s of the Candle and scalar matmul; checks they agree.

run flags

• --max-tokens <n> (default 128): maximum new tokens.
• --temperature <f> (default 0 = greedy), --top-k <n>, --top-p <f>, --seed <n> (default 0): sampling controls. Any non-greedy flag enables sampling; output is reproducible for a fixed seed.
• --tokenizer-json <path>: use this tokenizer instead of the GGUF one.
• --mem-budget <size> (for example 256MB, 1GiB): stream blocks under a hard byte budget instead of holding them resident.
• --explain-policy: derive the budget, prefetch depth, and thread count from the adaptive policy and print the reasoning.
• --threads <n|auto> (default auto): size the compute thread pool.
• --quantize-kv: store the key-value cache in f16 to extend context on tight RAM.

Generated text goes to stdout; timing (load, prefill / time-to-first-token, and decode tokens per second) goes to stderr, so output stays clean for piping.

Bindings

One engine, embeddable from other languages:

• C API: crates/allm-capi exposes a stable C ABI with a cbindgen-generated header (crates/allm-capi/include/allm.h). Panics are caught at the boundary and turned into status codes. See examples/c/main.c.

• Python: crates/allm-python is a PyO3 extension. pip install maturin, then maturin develop --release in that directory:

  import allm
  m = allm.Model("models/qwen.gguf")
  print(m.generate("The capital of France is", max_tokens=20))
  for piece in m.stream("Hello", max_tokens=20):
      print(piece, end="", flush=True)

Testing

cargo test            # unit + integration tests (no model needed)

The unit suite uses tiny synthetic models, so it is fast and needs no downloads. End-to-end tests that load a real GGUF model are gated on the ALLM_TEST_GGUF environment variable: they skip cleanly when it is unset (so CI stays green) and run for real when it points at a model:

# Linux / macOS
ALLM_TEST_GGUF=models/qwen.gguf cargo test -p allm-compute --test qwen_e2e --release

# Windows (PowerShell)
$env:ALLM_TEST_GGUF="models/qwen.gguf"; cargo test -p allm-compute --test qwen_e2e --release

These verify the tokenizer round-trips, that greedy decoding is deterministic, and that the streaming model reproduces the resident model's output token for token while respecting its hard memory budget.

Android and Termux

ALLM cross-compiles to aarch64-linux-android. See docs/ANDROID.md for the build setup, Termux usage, and memory guidance (notably: do not mlock on tight devices, and budget the context length).

Project layout

crates/
  allm-core        shared types, errors, logging
  allm-gguf        GGUF model file parser
  allm-tokenizer   tokenizer load and encode/decode
  allm-compute     ComputeBackend trait, Candle and scalar CPU backends, model
  allm-streaming   StorageBackend trait, mmap/direct readers, async prefetch
  allm-cache       LRU layer cache with a hard byte budget
  allm-adaptive    device probes and the streaming policy
  allm-scheduler   sessions, paged KV cache, continuous batching
  allm-capi        stable C ABI with a cbindgen-generated header
  allm-python      PyO3 extension module
cli/               the `allm` command line tool
docs/
  ARCHITECTURE.md  crate graph, traits, data flow, layerwise inference
  MEMORY_SEALING.md  the hard budget contract, layerwise inference, benchmarks
  ADAPTIVE.md      device probing, policy, runtime adaptation
  COMPARISON.md    feature matrix and measured numbers vs llama.cpp
  ANDROID.md       cross-compilation and Termux guidance
  NAMING.md        naming conventions

License

MIT © 2026 Mithun Gowda B — see LICENSE for details.