Home

TinyTitan Wiki

TinyTitan runs compatible Qwen-based MoE and dense text models locally on Apple Silicon using native Swift, Metal, and SSD-streamed mixture-of-experts weights. Because the experts are streamed rather than resident, it runs models far larger than the machine's RAM. It provides a CLI, an installer/repacker, and a loopback OpenAI-compatible server.

Important

TinyTitan serves text-only Qwen-based models at 4-bit and 8-bit: Qwen3.8-Flash-Next 125B-A6B, KAT-Coder-V2.5-Dev 35B-A3B, Qwen-AgentWorld 35B-A3B, Ornith 1.5 35B-A3B and Qwen 3.6 35B-A3B, plus the dense Qwen 3.5 2B/4B/9B on either engine. Ornith 1.5 8-bit is the default install. Vision is not included for any model; native MTP drafts are optional and experimental.

Start here

Nothing installed yet? Getting Started checks the Mac, installs the engine and a model, and runs a first prompt.

Already have a model? Three commands get you an answer:

tools/install_models.sh                                     # what is installed
.build/release/TinyTitanCLI --model models/qwen3.5_4B_4Bit \
  --prompt "The capital of France is" --max-new 32 --temperature 0
tools/server_launcher.sh --client server --model qwen35-4b --bits 4   # the API

The model directory is whatever tools/install_models.sh reports as installed. Generated text goes to stdout and timing to stderr, so a pipeline sees only the answer. Cookbook has a copy-paste recipe for each common task — serving the API, connecting a client, JSON output, memory, long context, the CPU engine, repairing a moved model — and says what you should see.

Last verified: 2026-09-28 by running these three commands on macOS 27.0 / Swift 6.4 / arm64. A --client codex|zed|… launch additionally rewrites that client's own provider config, so it was not run here.

Then, in this order:

1. Getting Started — requirements, install, first run.
2. Installation and Configuration — the model catalogue, environment variables, and every tool's own flags.
3. Cookbook — the task you are trying to do right now.
4. Start with 8-bit, standard responses, and Thinking off; change one control at a time with Runtime Controls.

Guides

Goal	Page
Install TinyTitan, or build it from source	Getting Started
Configure it: model installs, environment variables, launcher and tool flags	Installation and Configuration
Do a specific task, with copy-pasteable commands	Cookbook
Start the API, or connect a coding client or SDK	Local Server and API
Choose context, sampling, memory, thinking and engine settings	Runtime Controls
See supported capabilities and their limits	Features
Understand the runtime	System Design
Solve common problems	FAQ
Know what performance to expect	Benchmarks · Benchmarking Guide
Run several Macs from one place (DeepSeek Harness)	LAN Manager · TinyTitan Plugin
See what changed in each release	Changelog
Look at open work	Project Tracker

Requirements

• Apple Silicon (arm64), M1 or newer. Every published measurement comes from one M3, so the wider range is design intent rather than a tested result — nothing is known to be wrong on the other chips, and nothing has been run on them
• macOS 26 or later. Building from source additionally needs Xcode 27 with Swift 6.4 — the only supported toolchain
• SSD space for the installation: roughly 19.5–20.0 GB (35B 4-bit), 36.9–41.0 GB (35B 8-bit), about 162 GB for Qwen3.8-Flash-Next 4-bit, or about 220 GB for its 8-bit build — each carries its own hashed n-gram table, 95.4 GiB (102.4 GB) for the 125B
• RAM is not the limit that model size is: the 125B model runs on 24 GiB because only a bounded slice of its experts is resident
• One model process at a time

The server binds only to 127.0.0.1 and has no authentication or TLS. Never proxy, tunnel, or expose it to another host.