System Design

System Design

TinyTitan runs a model larger than the desired resident working set by separating always-needed tensors from routed mixture-of-experts weights.

Data flow

pinned checkpoint (safetensors)
      ↓ affine group-64 quantization   (skipped if already quantized)
      ↓ TinyTitanRepack
verified .gturbo directory
      ↓ model load
resident shared weights + bounded expert cache
      ↓ each token
Metal attention/router → selected expert reads → Metal MoE/head → token

Installation format

TinyTitanRepack converts a pinned Hugging Face checkpoint into a directory that contains aligned tensor data, a manifest, tokenizer assets, and a verified-install.json receipt. The receipt binds the payload to its absolute path. Moving the directory requires a local --verify-install pass, not a re-download.

What TinyTitan actually requires of a checkpoint

TinyTitan does not use MLX, and does not depend on an MLX release existing. It has no MLX dependency at runtime: inference is native Swift over hand-written Metal kernels, and the only package dependencies are a tokenizer and an HTTP server.

What it requires is a quantization layout — affine, group size 64, BF16 scale and bias, packed into u32 words. That happens to be the layout MLX writes, which is why an mlx-community release can be repacked directly and why the tensor names in this project mention MLX. Publishing in that layout is a convenience, not a precondition.

TinyTitanRepack is a repacker rather than a quantizer: it expects .weight already packed with companion .scales and .biases. The quantization step is separate and lives in this repository — quantize_affine() in tools/prepare_ornith_mtp.py, with a Swift equivalent in Quantization.quantizeInt4Affine. So an unquantized bf16 or fp16 checkpoint is a perfectly good source, and generally a better one: quantizing once from the original weights avoids inheriting somebody else's quantization error.

GGUF is a poor source by comparison. Its k-quants use super-blocks with a different structure, so there is no direct reinterpretation into affine group-64; converting means dequantizing and re-quantizing, which is lossy twice. Prefer the original weights where they exist.

The expensive part of adding a model is never the weight format. It is the architecture: a new family needs its config, its Metal kernels and a numerical parity pass against a reference implementation. A model that fits an existing family is mostly a conversion job; a new architecture is not.

Bounded expert streaming

Qwen 3.6 and Ornith 1.5 use 256 routed experts per layer and select eight for each token. Shared weights remain resident. Routed expert weights are loaded into a bounded per-layer cache only when selected.

Expert reads bypass the OS page cache by default, making the configured cache budget the actual model working set. Cache misses are filled in parallel. During decode, TinyTitan immediately executes phase 1 for cache hits while those miss reads are in flight, then runs a bounded fixup for the newly loaded experts. Slot leases prevent the GPU from observing an evicted or overwritten buffer. Decode speed can still be limited by scattered SSD reads when the active expert pool does not fit in memory.

Prefill and decode

• Prefill processes the prompt in chunks, builds model state, and produces the first-token state. Larger chunks reduce repeated expert sweeps but need more temporary GPU memory. Full-attention prefill blocks run on the Neural Engine from an exported Core ML sidecar — on by default since 4.6, with TINYTITAN_PREFILL_ANE=off opting out. A model with no sidecar falls back to the GPU quietly, and short prompts and decode stay on the GPU either way.
• Decode generates one token at a time. Each token runs the model's attention or gated-DeltaNet path, routes to the active experts, combines the shared and routed branches, evaluates the output head, and samples the next token with a tiled GPU Top-K reduction.
• KV state uses selectable 16-, 8-, or 4-bit storage (8-bit default) and grows from 8,192 tokens rather than reserving maximum context at load.

Native MTP

The optional Qwen/Ornith MTP sidecar implements the checkpoint's native one-layer draft: normalized next-token embedding plus target hidden state, 4096-to-2048 projection, one full-attention MoE layer, final norm, and the target's shared output head. The target verifies every proposed token. Draft routed experts stay SSD-resident behind their own fixed cache, and the draft KV/state allocation is included in a strict incremental memory budget.

The graph is shared between Ornith and Qwen 3.6 because they use the same Qwen3.5-MoE tensor contract; model IDs prevent cross-pairing their independently trained weights. Qwen3.8-Flash-Next has its own draft adapter, which fuses the target's wide hyper-connection residual with the next token's embedding rather than a 4096-to-2048 projection.

MTP is optional because the benefit depends on acceptance and hardware cost, and under SSD streaming it currently does not pay. On Qwen 3.6 35B-A3B a width-2 verify measures 12.68 experts a layer against 8.00 for a single token — 1.585x the experts — while accepting 57.4% of proposals, which is 1.574 tokens a pass. Cost 1.585 against benefit 1.574: they cancel, and every other per-pass overhead turns it into a net loss (about 0.85x end to end). Qwen3.8-Flash-Next is worse again, at roughly 2.8x against a 2.0x ceiling. The same technique wins on machines that hold the whole model in RAM, where a verify re-reads weights that are already resident.

Prompt-state reuse

The server can restore an exact compatible conversation prefix. Model snapshots include both full-attention KV rows and gated-DeltaNet recurrent state; restoring only KV would be incorrect. Entries are rejected when the model, runtime profile, context, template, tools, or prompt lineage differs.

RAM reuse is enabled by default. An optional private SSD tier survives server restarts. The cache stores inference state rather than completed responses.

Interfaces

The engine and its tools share one runtime and one model format:

• TinyTitanRepack: install, resume, discard, and verify models
• TinyTitanCLI: one-process command-line generation
• TinyTitanServer: one-model loopback OpenAI-compatible API
• TinyTitanBench: kernel and runtime measurements

Two more build from this repository but not from the engine: tinytitan-memory (with the ContinuityDemo harness beside it) is the persistent agent memory, which keeps its own store, and ttlanmanager drives a DeepSeek Harness fleet over the network. Neither loads a model or reads a .gturbo install.

Read Features for product capabilities and Runtime Controls for supported settings.

For the measurements and rejected approaches behind specific subsystems, see Technical Articles — engineering write-ups that are not required reading for using TinyTitan.