FAQ
Start with Ornith 1.5 8-bit using standard responses and Thinking off. It is the default installer, launcher, benchmark, and real-inference baseline because it passed every 8-bit cell in the current coding/tooling qualification; its fastest correct cell also beat the passing 4-bit cell. Choose Ornith 4-bit when about 19.5 GB storage and faster raw decode matter more than that result. Qwen 3.6 remains available explicitly.
No. Six-bit support was withdrawn in 3.9. Existing files are historical and not necessarily corrupt, but current installers and runtimes support 4-bit and 8-bit.
The routed expert pool is larger than the desired resident working set. TinyTitan keeps shared weights and a bounded expert cache resident, then reads only the experts selected for each token. This keeps memory use controlled at the cost of storage-dependent decode speed.
The prompt must be prefilled before decode begins. Large coding clients can send thousands of tokens of instructions and tools. Use prompt-state reuse for continued conversations, or the chat-only fast alias for direct questions.
ornith-1.5-35b-a3b_8-Bit-fast uses the same model weights but removes known coding-CLI
system prompts, tool definitions, and reminder scaffolding before prefill. It
can answer direct questions much sooner, but it cannot perform the stripped
agent tool loop. Use ornith-1.5-35b-a3b_8-Bit for tool-based work. A manually
served Qwen 3.6 model exposes the equivalent qwen3.6-35b-a3b_8-Bit[-fast] IDs.
Every id ends in the routed-expert width, so check /v1/models rather than
assuming the bare name.
The first request is a miss. A hit requires an exact compatible prefix and the
same model, runtime profile, context, template, and tools. RAM entries disappear
when the server restarts or unloads. Configure --prompt-cache-disk when
compatible state must survive restarts.
Sometimes, when the expert cache is too small. Beyond that point, more slots
add memory pressure without helping. The default is a shipped per-install entry,
measured on that install rather than derived by a rule: 10 GiB (160 slots) at
4-bit and 12 GiB (96 slots) at 8-bit for the 35B families, and 12 GiB (96
slots) at 4-bit, 9.5 GiB (40 slots) at 8-bit for Qwen3.8-Flash-Next. The 35B
families have flattened above a 90% hit rate, where extra slots buy little;
Qwen3.8 is still climbing at 96 because 512 experts at top-10 spread far wider
than 128 at top-8 — and on a 24 GB machine its 8-bit row ships the measured
middle of the range, 40 slots, because 48 is faster but grows swap run over run.
The budget is clamped to half of physical memory, so a smaller Mac falls back
rather than over-committing. Benchmark before changing it with --ram-budget.
No to both. TinyTitan has no MLX dependency: inference is native Swift over hand-written Metal kernels, and the package depends only on a tokenizer and an HTTP server.
What it needs is a quantization layout — affine, group size 64, BF16 scale and
bias, packed into u32. MLX happens to write exactly that, which is why an
mlx-community release can be repacked directly. It is a convenient supply of
checkpoints, not a requirement.
An ordinary bf16 or fp16 safetensors checkpoint works too, and is usually the
better source: quantizing once from the original weights beats inheriting
somebody else's quantization error. The quantizer is in this repository
(quantize_affine() in tools/prepare_ornith_mtp.py). GGUF is the weakest
source, because its k-quants would have to be dequantized and re-quantized —
lossy twice.
The real cost of adding a model is the architecture, not the file format. A model that fits a family TinyTitan already implements is largely a conversion job; a new architecture needs kernels and a numerical parity pass.
It stores and reads more expert data per generated token. Its value is higher weight fidelity, not throughput.
Enable YaRN with --rope-scaling yarn. It selects 1M by default; add
--max-context 524288 for 512K. Native RoPE remains capped at 262,144 tokens.
The KV cache defaults to 8-bit and is independent of the installed model's
weight precision. At 1M, use --kv-bits 4 on a memory-constrained Mac. YaRN
cannot currently be combined with MTP. See Runtime Controls
for commands and memory estimates.
Usually not. The receipt is bound to the original absolute path. Reissue it in place:
swift run -c release TinyTitanRepack \
--verify-install \
--input-gturbo /new/path/to/modelDo not edit the receipt manually.
Do not run multiple model processes. Check with:
pgrep -fl 'TinyTitanServer|TinyTitanCLI|TinyTitanPackageTests|swiftpm-testing-helper|mlx_lm|mlx-lm'Stop only a process you launched.
No. It can generate function-call requests, but the client decides whether to run them and applies its own permission policy.
No. It has no remote authentication or TLS and must remain on 127.0.0.1.
Do not proxy or tunnel it.
Yes, for text generation and function calls in 4-bit or 8-bit. TinyTitan imports the pinned official Ornith MLX text checkpoint into the same bounded, SSD-streamed Qwen3.5-MoE runtime used by Qwen 3.6. Ornith's own tokenizer and chat template are installed with the model. Vision is not supported. Ornith's native MTP layer can be prepared as an experimental sidecar, but measured M3 decode did not improve, so MTP is disabled by default. Qwen 3.6 remains available.
For installation problems, return to Getting Started. For server setup, use Local Server.
Start
Use TinyTitan
DeepSeek Harness
Reference
Engineering
Project