Runtime Controls

Runtime Controls

Start with the defaults. Change one setting at a time and reload the model when changing a load-time control.

Generation

Control	CLI	Default	Use
Response limit	--max-new	1,024	Maximum generated tokens
Context scaling	`--rope-scaling none	yarn`	none
Context	--max-context	Native: 256K (CLI and server); YaRN: 1M	Prompt plus response capacity
KV cache	`--kv-bits 16	8	4`
Temperature	--temperature	0.6; Qwen3.8: 1.0 thinking / 0.7 non-thinking	0 is greedy; positive values sample. A family whose model card specifies otherwise overrides the house value
Top-K	--top-k	20	Keeps at most K candidates; 0 disables
Top-P	--top-p	0.95; Qwen3.8 non-thinking 0.80	Nucleus truncation
Presence penalty	API presence_penalty	0.0; 1.5 on Qwen3.8 when thinking is off	Subtracted once per distinct id already in the history
Repetition penalty	--repetition-penalty	1.0	Penalizes repeated tokens
Seed	--seed	Off	Reproducible sampling
Stop text	--stop	None	Repeat the flag for multiple strings
Concise answers	--concise	Off	Adds the shorter-answer system prompt
Thinking	`--thinking off	on`	Off

Native context supports up to 262,144 prompt-plus-response tokens. Enable YaRN to select 524,288 or 1,048,576 tokens; enabling it without --max-context selects 1M. Native RoPE remains the default.

The sampling defaults are per family, and Qwen3.8-Flash-Next has two rows, because its card specifies different values inside and outside thinking mode. The row is chosen from the mode the request actually runs in, at both 4-bit and 8-bit:

family / mode	temperature	top-p	top-k	min-p	presence	repetition
Qwen3.8 — Thinking (default)	1.0	0.95	20	0.0	0.0	1.0
Qwen3.8 — Instruct / non-thinking	0.7	0.80	20	0.0	1.5	1.0
Qwen 3.5 and 3.6, Ornith 1.5, AgentWorld	0.6	0.95	20	0.0	0.0	1.0
KAT-Coder-V2.5-Dev	1.0	0.95	20	0.0	0.0	1.0

Min-p is 0.0 in every row and the filter is not implemented: the runtime refuses a non-zero value rather than sampling as if it were zero. Presence penalties are implemented — the Qwen3.8 instruct row is the only row that uses one, and before 2026-09-21 the engine refused any non-zero value.

These are gap-fillers, not overrides: an explicit --temperature or a request field always wins.

KV storage starts at 8,192 tokens and grows on demand. For the production Qwen/Ornith topology, full-length attention KV uses approximately:

Context	16-bit KV	8-bit KV	4-bit KV
512K	10.0 GiB	5.31 GiB	2.81 GiB
1M	20.0 GiB	10.63 GiB	5.63 GiB

Use 4-bit KV when 1M must fit on a 24 GB Mac. The model weights keep their installed 4-bit or 8-bit format; --kv-bits changes only the live attention cache. YaRN changes RoPE extrapolation and can affect quality beyond the native window. It is not currently compatible with MTP.

Examples:

# Native context with the default 8-bit weights and KV cache
.build/release/TinyTitanCLI --model models/ornith-1.5_35B_A3B_8Bit \
  --prompt "Summarize this code" --max-context 262144

# 1M YaRN with 4-bit KV; omit --max-context because 1M is the YaRN default
.build/release/TinyTitanCLI --model models/ornith-1.5_35B_A3B_8Bit \
  --prompt "Summarize this code" --rope-scaling yarn --kv-bits 4

# Explicit 512K YaRN
.build/release/TinyTitanCLI --model models/ornith-1.5_35B_A3B_8Bit \
  --prompt "Summarize this code" --rope-scaling yarn \
  --max-context 524288 --kv-bits 8

Model loading and I/O

Control	Values	Default	Notes
Server expert RAM	--ram-budget 8G or another positive size	Per family (see below)	Overrides the tuned default; the runtime otherwise derives slots from the family and expert stride
Expert slots	--expert-cache-slots = 8, 16, 24, 32, 40, 48, 64, 96, 112, 128, 160, 192, 256	Derived	Server and CLI both derive from the budget; an explicit count overrides it. Both --help texts print that same list
Launcher RAM limit	tools/server_launcher.sh --ram <GB>	Model default	Any positive whole GB, with or without a G; the interactive menu offers 1, 2, 4, 8, 16 and 32. Convenience over --ram-budget; the default keeps the install's measured profile. CPU models have no expert cache and ignore it
Prefill chunk	--prefill-chunk = 32…4096; CLI also accepts auto	Supported 35B models: 4096	Larger chunks can reduce repeated expert reads but use more temporary memory
Read advice	`--rdadvise off	default	bounded
Bounded expert I/O	TINYTITAN_BOUNDED_IO=0 opts out	On	Bypasses the OS page cache so the configured budget is real. Opting out measured about +4-5% decode on Qwen3.8-Flash-Next 4-bit (3/3 and 2/2 paired rounds on a short and a ~500-token prompt, output identical) but gives nothing for prefill and grows swap. Do not use it: the declared RAM budget is a product guarantee, and the decision is that it stays enforced — the footprint trade is not available at any throughput. Read advice earns its keep only in this mode
Parallel expert fills	TINYTITAN_PARALLEL_IO=0 disables	On	Fills cache misses concurrently
Decode expert execution	TINYTITAN_DECODE_EXPERT_EXECUTION=hit-fixup|barrier|gpu-residency	hit-fixup	Overlaps cached expert work with miss reads; barrier is the deterministic comparison path. gpu-residency keeps a layer's experts resident on the GPU, and a profile with early expert hits can select it on its own
Expert I/O backend	`TINYTITAN_EXPERT_IO_BACKEND=pread	metal`	pread
Sampler path	`TINYTITAN_SAMPLER_PATH=tiled	generic`	tiled
MTP verify schedule	`TINYTITAN_MTP_VERIFY=pair	tile`	pair
ANE prefill	TINYTITAN_PREFILL_ANE=off|on	on	Full-attention prefill on the Neural Engine from an exported sidecar, 2.31x prefill. Falls back to the GPU when no sidecar is present. Output is not byte-identical to the GPU path. Keep one model resident: the Core ML arenas evict the expert slot cache
Speculative expert prefetch	TINYTITAN_PREDICTIVE_PREFETCH=0|1, TINYTITAN_PREFETCH_TOP_M=1…topK	Per family (see below)	Reads the next layer's predicted experts during the current layer. Forces on or off in either direction

Per-family defaults

Both the expert-cache budget and the prefetch switch are governed by the same quantity: how much of the token is spent on expert I/O. That varies by nearly 3x across the shipped families, so neither is one number.

Family	Expert RAM	Prefetch	Measured
Qwen3.8-Flash-Next 4-bit	12 GiB (96 slots; clamped to 8 GiB / 64 slots on a 24 GiB Mac)	depth 1	3.993 -> 4.621 tok/s (+15.7%) on a 7-token prompt and 3.627 -> 4.158 (+14.6%) on a ~500-token one, output byte-identical
Qwen3.8-Flash-Next 8-bit	9.5 GiB (40 slots)	depth 1 (inferred)	the no-paging middle of the measured range: 32 slots (8 GiB) is slower, 48 (13 GiB) faster but grows swap run over run. The depth is inherited from the 4-bit A/B, not measured here
Qwen 3.6 / Ornith 8-bit	12 GiB (96 slots)	depth 1	+11.3% / +12.6%
Qwen 3.6 / Ornith 4-bit	10 GiB (160 slots)	depth 1	+1.8% / +1.8%

Prefetch pays at both widths once the ring is one read deep, and it reads the next layer's predicted experts during the current layer, so those reads land before demand rather than after the next plan: it turns speculative reads into hits instead of merely warming pages. It measured as a regression at 4-bit while the ring was clogged (about five reads issued per token), and the 2026-09-05 session's 512-token story runs again preferred prefetch off; re-measured on the current engine (2026-09-21) it wins at both prompt lengths tried. Depth is sharply peaked at that single read in flight: depth 2 gives about +2.7% where depth 1 gives +9.8%.

The tuned budget is clamped to a third of physical memory, so this 24 GiB Mac runs Qwen3.8-Flash-Next 4-bit at 8 GiB / 64 slots rather than the row's 12 GiB. That clamp is also the measured optimum: a --ram sweep from 1 to 16 GB (short prompt, 256 tokens, 2026-09-21) peaks at 64 slots, 4.55-4.60 tok/s, while 96 slots gives 4.00 and 128 gives 2.26 — even though the hit rate climbs 0.757 -> 0.866 and expert reads fall 80.8 -> 44.5 GiB, because the extra wait (84 -> 297 ms) and swapping cost more than the reads save. 1 GB is not slow but unusable: the model routes top-10 and 8 slots cannot hold ten experts, so generation fails outright; 2 GB / 16 slots (2.87 tok/s) is the floor.

Smaller expert budgets save memory but can slow decode because more routed experts must be read from SSD. Larger budgets are not automatically faster; measure on the target Mac. Hit/fixup changes scheduling, not the RAM budget: in-flight cache slots are leased until their Metal command completes and cannot be evicted or reused early.

Server controls

Flag	Default	Purpose
--rope-scaling	none	Select native RoPE or YaRN
--kv-bits	8	Select 16-, 8-, or 4-bit KV storage
--thinking	off	Select the model's binary reasoning mode
--mtp-model	Off	Native Qwen/Ornith MTP sidecar; greedy and native RoPE only
--mtp-memory-mib	384	Strict incremental MTP budget, 256–512 MiB
--queue-limit	4	Maximum queued requests behind the concurrent width
--max-concurrent-sequences	1	Generations served at once: a power of two up to 256. Above 1 each running sequence keeps its own KV cache, so memory use rises and one shared GPU makes every answer slower — it buys fairness, not throughput, and the prompt cache is off above 1. The width actually built is clamped to what memory allows, and the server logs it
--prompt-cache-mode	multi-prefix	off, single-prefix, or multi-prefix state reuse
--prompt-cache-entries	4	Retained prefix count
--prompt-cache-memory-mib	256	RAM snapshot budget
--prompt-cache-disk	Off	Private persistent cache directory
--lazy-load	Off	Defer model load until the first inference request
--idle-unload-seconds	0	Unload after an idle period; implies lazy loading

Run .build/release/TinyTitanServer --help for the complete accepted ranges.

Answer modes

Concise mode shortens answers by adding a terse system prompt. It changes answer length, not decode tokens per second. If important nuance is omitted, turn it off.

Surface	Control
CLI	--concise
Server	TINYTITAN_CONCISE_MODE=1
Launcher	default selects standard responses; pass concise as the 4th argument to opt in

Thinking mode opens the model's reasoning branch. Thinking can add substantial output and wall time. Every TinyTitan surface defaults to off.

What each family actually renders:

Family	Levels
Ornith 1.5, Qwen 3.6, AgentWorld, KAT-Coder-V2.5-Dev, Qwen 3.5 dense	off, on — the binary switch
Qwen3.8-Flash-Next	off, low, medium, xhigh (template default xhigh)

Surface	Control
CLI	`--thinking off
Server	`--reasoning <off
Launcher	off, on, or any level the chosen model lists
Benchmarks	`TINYTITAN_THINKING_MODE=off

Reasoning requests are best-effort, never fatal

A server-wide level is also the level a request may name, and coding agents send vocabularies this project never defined — xhigh on an on/off model, high on a model with only low/medium/xhigh, ultra, none, extra-high. Those used to be a 400, which ended an agent's turn mid-work. They are now mapped onto the nearest level the served model can render, the request is served, and the server logs what it applied:

request ornith-1.5-35b-a3b_8-Bit reasoning: reasoning level 'xhigh' is not
supported by this model; applied on instead (supports: off, on)

Ties go to the cheaper level, the same rule the server-wide --reasoning uses, so a request and a launch flag cannot disagree. A level that means nothing on any ladder is also not fatal: the model's own default applies and the log says it was not recognised.

Mid-session switches

A request that names a level the model does not offer is mapped, as above. A request that names one it does offer is honoured, and that includes changing it while the session is live: the level takes effect on the very next request, with no restart.

This matters most for turning thinking off. A coding agent that finds a model over-thinking a simple turn sends reasoning_effort: "off" and gets a direct answer from the same server, then turns it back on for the next hard question. Both directions work — off→on and on→off — and so does moving between effort levels on a model that defines them.

Mechanically, thinking and effort are baked into a tokenizer when it loads, so a switch resolves a tokenizer for the requested configuration rather than reusing the loaded one. That tokenizer also carries the think-block and stop token IDs for its own mode, so decode follows the switch and not only the prompt text. The shared (folder, thinking, effort) cache makes a repeat switch a lookup rather than a load.

Three consequences worth knowing:

• A switch re-prefills. The prompt renders differently, so it is a cache miss and the first request after a switch pays full prefill. Stable levels keep every cache benefit; this is only the cost of changing one.
• An unrecognised level is not a switch. A word that means nothing on the ladder leaves the loaded level in place, rather than guessing, and says so in the log.
• A model can think anyway. The switch is a prompt-side instruction, and one shipped install ignores it: Qwen AgentWorld 35B-A3B 8-bit, asked a false-premise question with thinking off, opens a reasoning block of its own and spends the whole max_tokens inside it (measured; the 4-bit install of the same family answers in 8 tokens). That text is reasoning_content, never content, and the server logs thinking off, but the model wrote N characters of reasoning on the request's line. So a capped request that comes back with an empty content has an answer in the log: raise max_tokens or turn thinking on. The measurements are in One Prompt, Every Model.

Practical presets

• General launcher use: Ornith 8-bit, full, concise off, native 262K context, multi-prefix cache, 8-bit KV, and MTP off.
• Direct coding-CLI questions: 8-bit, fast, optionally concise and thinking off.
• Agent tool loops: base model (full), not the fast alias.
• Shared Mac: server with --lazy-load --idle-unload-seconds N and a private disk prompt cache.
• Benchmark: fixed prompt, seed, context, and controls; see Benchmarking Guide.