Home

TinyTitan Wiki

TinyTitan runs compatible Qwen-based MoE and dense text models locally on Apple Silicon using native Swift, Metal, and SSD-streamed mixture-of-experts weights. Because the experts are streamed rather than resident, it runs models far larger than the machine's RAM. It provides a CLI, an installer/repacker, and a loopback OpenAI-compatible server.

Important

Current installers support text-only Qwen-based models at 4-bit and 8-bit: Qwen3.8-Flash-Next 125B-A6B, KAT-Coder-V2.5-Dev 35B-A3B, Qwen-AgentWorld 35B-A3B, Ornith 1.5 35B-A3B and Qwen 3.6 35B-A3B, plus the dense Qwen 3.5 2B/4B/9B on either engine. Ornith 1.5 8-bit is the default install and the default golden target. Six-bit support was withdrawn in 3.9. Vision is not included for any model; native MTP drafts are optional, experimental, and disabled by default.

Start here

Nothing installed yet? Getting Started checks the Mac, builds, and installs a model. It is the only page you have to read in order.

Already have a model? Three commands get you an answer:

tools/install_models.sh                                     # what is installed
.build/release/TinyTitanCLI --model models/qwen3.5_2B_4Bit \
  --prompt "The capital of France is" --max-new 32 --temperature 0
tools/server_launcher.sh --client codex --model qwen38 --bits 4   # API + Codex

Generated text goes to stdout and timing to stderr, so a pipeline sees only the answer. Cookbook has a copy-paste recipe for each common task — serving the API, connecting a client, JSON output, memory, long context, the CPU engine, repairing a moved model — and says what you should see.

Then, in this order:

1. Getting Started — requirements, install, first run.
2. Cookbook — the task you are trying to do right now.
3. Start with 8-bit, standard responses, and Thinking off; change one control at a time with Runtime Controls.

Guides

Goal	Page
Install and run TinyTitan	Getting Started
Do a specific task, with commands	Cookbook
See supported capabilities	Features
Start the API or a coding CLI	Local Server
Choose context, sampling, and memory settings	Runtime Controls
Solve common problems	FAQ
Understand the runtime	System Design
Read or reproduce measurements	Benchmarks · Benchmarking Guide
Review releases	Changelog

The Engineering and Project sections in the navigation are for people building on or contributing to TinyTitan — deep-dive articles and the project tracker. Nothing there is needed to use it, and nothing in them is a promise or a supported feature.

Requirements

• Apple Silicon (arm64), M1 or newer. Every published measurement comes from one M3, so the wider range is design intent rather than a tested result — nothing is known to be wrong on the other chips, and nothing has been run on them
• macOS 26 or later
• Swift 6.4 or later
• SSD space for the installation: roughly 19.5–20.0 GB (35B 4-bit), 36.9–41.0 GB (35B 8-bit), about 162 GB for Qwen3.8-Flash-Next 4-bit, or about 220 GB for its 8-bit build — each carries its own hashed n-gram table, 95 GB on its own
• RAM is not the limit that model size is: the 125B model runs on 24 GiB because only a bounded slice of its experts is resident
• One model process at a time

The server binds only to 127.0.0.1 and has no authentication or TLS. Never proxy, tunnel, or expose it to another host.