Cookbook

Cookbook

Copy-paste recipes for the things people actually do, each with what you should see. They assume you have followed Getting Started — the Mac is checked, the release build exists, and at least one model is installed.

Tip

One model process at a time. Before any run below, this should print nothing:

pgrep -fl 'TinyTitanServer|TinyTitanCLI|TinyTitanPackageTests|swiftpm-testing-helper|mlx_lm|mlx-lm'

If it prints a process you did not start, stop and ask rather than killing it.

Run a single prompt

.build/release/TinyTitanCLI \
  --model models/qwen3.5_2B_4Bit \
  --prompt "The capital of France is" \
  --max-new 8 \
  --temperature 0

 Paris.
A. True
B

The generated text goes to stdout; the timing footer goes to stderr, so a pipeline sees only the answer:

[stop=maxTokens prefill=5tok/0.10s new=8tok decode=0.13s tok/s=61.8]

Point --model at any install under models/, and see everything the CLI takes with .build/release/TinyTitanCLI --help.

See what is installed

tools/install_models.sh

MODEL                WIDTH    STATE      SOURCE
ornith15             4-bit    -          convert_qwen35moe
qwen38flash          4-bit    installed  convert
qwen38flash-8bit     8-bit    installed  convert
katcoder             4-bit    -          convert_qwen35moe
agentworld           4-bit    installed  convert_qwen35moe
qwen35-2b            4-bit    installed  convert_qwen35
qwen35-2b-8bit       8-bit    installed  convert_qwen35

tools/install_models.sh <name>          install one width
tools/install_models.sh <name> both     4-bit and 8-bit from one download
tools/install_models.sh --help          sources and disk sizes

STATE is installed or -. The keys in the first column are what tools/install_models.sh <key> accepts. The launcher takes the base family keys only — ornith, qwen36, agentworld, katcoder, qwen38, qwen35-2b, qwen35-4b, qwen35-9b — and selects the width with --bits 4|8, so a key like qwen38flash-8bit installs but is not a launcher key.

Install a model

tools/install_models.sh katcoder both      # both widths, ONE download
tools/install_models.sh                   # re-run with no argument to confirm

Two widths of the same checkpoint convert from a single fetch, so both is cheaper than installing each width separately. An interrupted download continues where it stopped:

swift run -c release TinyTitanRepack \
  --model ornith15-8bit \
  --output models/ornith-1.5_35B_A3B_8Bit \
  --resume

Models are large — roughly 19.5 GB for a 35B 4-bit install and about 162 GB for Qwen3.8-Flash-Next 4-bit — so check free space first. See Getting Started for the full catalogue and the MTP sidecar.

The Qwen3.8 conversion itself resumes as well, and at the shard level: an interrupted run keeps the output shards it finished and fetches only what is missing, so a dropped connection costs the shard in flight rather than the 360 GB. To fetch through a mirror, set the Hub's own variable (the converter also takes --endpoint):

HF_ENDPOINT=https://hf-mirror.com tools/install_models.sh qwen38flash

A run stopped between finishing its shards and writing the index refuses to convert into that directory again — that would leave two copies of a 360 GB snapshot — so delete the directory or point the tool at a new one. Mirrors are trusted as given: nothing checks a shard against a published hash.

Start the API and call it

swift build -c release --product TinyTitanServer
.build/release/TinyTitanServer \
  --model models/qwen3.5_2B_4Bit \
  --port 8083

Keep that terminal open. From another one:

curl --silent http://127.0.0.1:8083/health
curl --silent http://127.0.0.1:8083/v1/models

curl --silent http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.5-2b_4-Bit",
    "messages": [{"role": "user", "content": "Reply with exactly READY."}],
    "temperature": 0,
    "max_completion_tokens": 16
  }'

Take the model id from /v1/models rather than typing it: it always ends in the routed-expert width (_4-Bit, _8-Bit), and a dense install also offers a @cpu alternative. The server is loopback-only and has no authentication — never proxy or expose it. Local Server covers the full API, streaming, prompt reuse and tool calls.

Start the API and a coding client together

tools/server_launcher.sh --client codex --model qwen38 --bits 4
tools/server_launcher.sh --client zed --model qwen38 4 --ram 12
tools/server_launcher.sh                       # ask me everything
tools/server_launcher.sh --client server --port 9123   # a port of your own

The launcher starts the server and wires the chosen client's provider config to it, on 127.0.0.1:8080 unless you answer the port question otherwise (--port or TINYTITAN_PORT skips the question). Clients: server, codex, claude, qwen, opencode, zed. --ram caps the expert cache in GB; anything over 30% of physical memory is warned about and used anyway, and the default is the install's own measured profile. Every other installed model stays reachable by name, one resident at a time.

Switch models, or thinking, without restarting

Name a different model in the next request — the server unloads one and loads the other:

curl --silent http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.5-4b_4-Bit",
       "messages": [{"role": "user", "content": "hi"}],
       "reasoning_effort": "off"}'

Thinking and effort switch per request in both directions; a switch re-prefills, because the prompt renders differently. See Runtime Controls.

Run a dense model on the CPU

The dense Qwen 3.5 installs run on either engine. Add @cpu to the id from /v1/models:

curl --silent http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.5-2b_4-Bit@cpu",
       "messages": [{"role": "user", "content": "hi"}]}'

Only the dense models have this choice — the MoE families stream experts and are GPU-only.

Ask for JSON

curl --silent http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.5-2b_4-Bit",
       "messages": [{"role": "user", "content": "The colour red, as JSON."}],
       "response_format": {"type": "json_object"}}'

The server compiles the schema into a grammar that masks the sampler, so the bytes that come back are a well-formed document of the requested shape — it constrains the shape, never the truth of the content. A named schema uses {"type": "json_schema", …}; the supported subset is small and explicit, and anything outside it is refused by name (docs/structured-output.md).

Give the model memory

TINYTITAN_MEMORY=1 .build/release/TinyTitanServer \
  --model models/qwen3.5_2B_4Bit --port 8083

Memory is scoped per workspace and outlives the conversation. The workspace defaults to default: the launcher scripts export TINYTITAN_WORKSPACE_DIR="$PWD", so a launch through them is scoped to the working directory, while the bare server command above writes to the shared default store under ~/.tinytitan/memory. TINYTITAN_MEMORY_TOOLS=full exposes the six memory_* functions to the model, and TINYTITAN_MEMORY_DIR moves the store. Off by default. See docs/agent-memory.md.

Repair a model directory that moved

Moving or renaming an install makes it fail with trusted install receipt invalid: model directory mismatch. The payload is fine — re-issue the receipt in place:

swift run -c release TinyTitanRepack \
  --verify-install \
  --input-gturbo models/qwen3.5_2B_4Bit

It re-hashes the payload and rebinds it, with no download. Never hand-edit verified-install.json: the path binding is what detects a moved or swapped directory.

Measure your own machine

Build once, then follow the Benchmarking Guide exactly — it fixes the prompt, seed, context and controls so two runs are comparable. The guided starting points are python3 benchmark/coder_cli_benchmark.py --round clients (checks every client's wiring without loading a model) and the scripts that page names. Numbers from one Mac do not transfer to another; measure rather than assume.

Where to go next

To do this	Read
Choose context, sampling, KV precision, thinking	Runtime Controls
Connect Codex, Claude Code, Qwen Code, OpenCode or Zed	Local Server
Understand why it is fast, and what is not measured	System Design, Benchmarks
Fix something that is not behaving	FAQ
See the engineering behind a subsystem	Technical Articles