OpenAI Compatible Server

Local OpenAI-Compatible Server

TinyTitanServer serves one installed model to local OpenAI-compatible clients. It binds to 127.0.0.1 without authentication or TLS.

Caution

Keep the server local. Do not proxy, tunnel, or expose it. Before starting, verify that no other TinyTitan or local-model process is running.

Start the server

memory_pressure -Q
pgrep -fl 'TinyTitanServer|TinyTitanCLI|TinyTitanPackageTests|swiftpm-testing-helper|mlx_lm|mlx-lm'

Continue only when the process check prints nothing.

swift build -c release --product TinyTitanServer
.build/release/TinyTitanServer \
  --model models/ornith-1.5_35B_A3B_8Bit \
  --port 8083

Keep that terminal open. Stop the server with Control-C.

Check it from another terminal:

curl --silent http://127.0.0.1:8083/health
curl --silent http://127.0.0.1:8083/v1/models

Send a request:

curl --silent http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ornith-1.5-35b-a3b_8-Bit",
    "messages": [{"role": "user", "content": "Reply with exactly READY."}],
    "temperature": 0,
    "max_completion_tokens": 16
  }'

The server derives ornith-1.5-35b-a3b_8-Bit from the verified manifest unless --model-id overrides it. KV cache precision defaults to 8-bit.

The model id always ends in the routed-expert width (_4-Bit, _8-Bit). The width is read from the manifest's routed-expert slot rather than parsed from the name, so a 4-bit and an 8-bit install of the same weights are distinguishable in /v1/models instead of both answering to one id. This is a breaking change: the bare ornith-1.5-35b-a3b is no longer accepted. Ask the server rather than assuming -- curl -s http://127.0.0.1:8083/v1/models lists exactly what it serves.

An optional prepared Ornith MTP sidecar can be supplied with --mtp-model. It requires greedy requests, native RoPE, and prompt cache off. MTP is experimental and disabled by default because current M3 measurements show no speed benefit; preparation is documented in Getting Started.

To enable extended context, choose YaRN and optionally override its 1M default with 512K. Use 4-bit KV when memory is tight:

.build/release/TinyTitanServer \
  --model models/ornith-1.5_35B_A3B_8Bit \
  --port 8083 \
  --rope-scaling yarn \
  --kv-bits 4

See Runtime Controls for the 512K form and KV memory table.

Supported API

Endpoint	Purpose
GET /health	Process health; never loads a lazy model
GET /v1/models	Lists the served model(s): the one model, or every install under --models-dir (plus <id>@cpu where a second engine is a real choice). In single-model mode the -fast alias is listed beside the model; under --models-dir only the real installs are listed, and -fast is accepted but hidden
POST /v1/chat/completions	Chat Completions, JSON or SSE streaming
POST /v1/responses	Responses API, JSON or SSE streaming
POST /v1/responses/compact	Compaction: a conversation in, a compacted input window out, which the next response takes as its base input
POST /v1/messages	Anthropic Messages, JSON or SSE streaming; selected by the anthropic-version header
POST /v1/messages/count_tokens	Anthropic token counting; never generates
POST /v1/models/unload	Releases a dynamically managed model

The server supports system, developer, user, assistant, and tool messages; sampling controls; stop strings; seeds; usage reporting; function tools; and enforced structured output: asking for JSON — response_format on Chat Completions, text.format on Responses, output_config.format on Messages — compiles the schema into a byte-level grammar that masks the sampler, so the model can only emit a document the schema allows. It does not provide embeddings, multimodal input, legacy Completions, batching, or log probabilities. Serving a catalog (--models-dir) does switch the resident model when a request names another of the listed installs — one at a time, waiting for in-flight generations first — and a single-model server refuses any name but its own. Thinking is a per-request control on all three surfaces — reasoning_effort on Chat Completions, the nested reasoning.effort on the Responses API, and thinking on the Messages API — so a session may think on one turn and answer directly on the next; the thoughts come back as reasoning_content (OpenAI) and as a thinking block with an empty signature (Messages). Thinking is off for a request that names a JSON format, because the grammar constrains every token.

The schema subset is small and explicit, and everything outside it is refused by name at request time rather than accepted and quietly ignored: Structured output lists the supported keywords, the mechanism, and what the guarantee does not cover (a response truncated by max_tokens is a truncated document).

Launcher

One script starts everything: tools/server_launcher.sh.

Run it with no arguments and it asks, in plain language and with a default on every question:

1. What to launch — the API server on its own, or the server plus Codex, Claude Code, Qwen Code, OpenCode or the Zed editor. The server speaks OpenAI and Anthropic at once, so this choice selects the client setup to write, not what the server runs.
2. Full agent loop or fast chat — fast strips coding-CLI boilerplate before prefill and serves the model's <id>-fast alias.
3. Which model, from the server's own catalog: every installed model and quantization, with its size, the engine that serves it and the thinking levels it renders. Only installs that are really under models/ are offered — the launcher re-checks each entry's directory, and when it has to fall back to its built-in list (no built server binary, no python3) that list is filtered the same way and the families it leaves out are named in one line. A model or a width that is not installed is never a choice, and asking for one is refused with the widths that are there. For almost every install the engine is a property rather than a choice: every 35B-A3B / 125B-A6B family is GPU-only, because the CPU engine does not implement those. The dense Qwen 3.5 2B / 4B / 9B installs (4- and 8-bit) are the exception -- both engines implement their family (qwen3_5_dense), so they show GPU+CPU and the engine is asked for, with the GPU as the default.
4. Answer style — standard or concise.
5. Reasoning level — only the levels the chosen model's template implements, off first and default. The dense Qwen 3.5 models define the binary enable_thinking switch, so their levels are exactly off (a direct answer) and on (reason before answering); Qwen3.8-Flash-Next renders off, low, medium, xhigh. A level the model does not define is refused rather than mapped to a neighbour, and a client can still switch it per request — reasoning_effort (or chat_template_kwargs) on Chat Completions, reasoning.effort on the Responses API, thinking on the Messages API, where disabled is a real off and an enabled block's budget_tokens picks the rung (<4096 low, <16384 medium, else xhigh). The level travels into generation, so a client may think on one turn and answer directly on the next.
6. RAM limit for the expert cache — 1, 2, 4, 8, 16 or 32 GB, or the model's own measured profile (the default). The question is built around 30% of the Mac's physical memory — the cache is wired, so it cannot be paged out and macOS, the KV cache, the prompt cache and everything else the person is running have to fit beside it. On a 24 GB Mac that is 7 GB, on a 16 GB Mac 4 GB, on an 8 GB Mac 2 GB. A --ram above it is allowed and warned about, in red, before the model starts: system instability while the model is loaded, heavy swapping that stalls other apps, and much slower generation, because a paging cache loses more than the extra slots gain. The warning is the operator's call to make, not a refusal, and the summary line keeps the fact visible. The default is the install's own measured profile, which the runtime separately holds to half of physical memory — measured on a 24 GB Mac with Qwen 3.8 4-bit, that machine's 12 GB profile left 11% of memory free and glitched audio. That is why the launcher's own rule is the tighter one, and why it is 30% rather than 40%: real usage ran past half of physical memory even with the cache at 40%, so the recommendation that overshot had to move. A CPU model has no routed-expert cache, so it does not ask: the step is skipped, and --ram, --kv, --context and --yarn are reported as not applying rather than passed along or dropped in silence.

The same choices can be passed positionally or as flags:

tools/server_launcher.sh [<client>] [fast|full] [<model> [4|8]] \
                         [default|concise] [<thinking>] [<ram>]

tools/server_launcher.sh --client codex --model ornith 4 --ram 8
tools/server_launcher.sh --client zed   --model qwen38 4 --context 524288 --yarn
tools/server_launcher.sh --model qwen35-2b --bits 4 --thinking on --engine cpu

--client server (or server/ornith 4 positionally) starts the API alone. Flags: --client, --model, --bits, --engine, --mode, --answers, --thinking, --ram, --context, --kv, --yarn, --port, --memory, --dry-run, --help. --engine cpu|gpu picks the engine for the one install shape that both engines serve (the dense Qwen 3.5 models); for every other install it asserts the single engine that implements the family and refuses the other with that family as the reason.

On a running server the same choice is a model-id suffix. The catalog id is the GPU spelling, and <id>@cpu / <id>@gpu name an engine explicitly, so a client selects one per request:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.5-2b_4-Bit@cpu", "messages": [{"role": "user", "content": "hi"}]}'

/v1/models lists the bare id and the @cpu alternative for a dense install (and only an alternative that is a real choice -- a single-engine install gets no second entry). Residency is still one model at a time, so switching a dense install between engines reloads it. Starting a dense install directly works too, without the catalog:

swift run -c release TinyTitanServer --model models/qwen3.5_2B_4Bit --cpu --reasoning on

What it does per client:

Client	Configured by
Codex	~/.codex-tinytitan/config.toml (a dedicated home; wire_api = "responses")
Claude Code	ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_*_MODEL in the environment
Qwen Code	~/.qwen-tinytitan/settings.json, with its stream timeouts disabled
OpenCode	an tinytitan provider merged into ~/.config/opencode/opencode.jsonc
Zed	an openai_compatible provider merged into ~/.config/zed/settings.json

Both JSONC merges keep every other key in the file, tolerate comments and trailing commas, and write a .tinytitan-backup beside the original before they rewrite it. A file they cannot parse is left untouched.

Server profile: every installed model is reachable by name (--models-dir, one resident at a time); GPU models are pinned to native 262,144-token context, a 256 MiB multi-prefix prompt cache, 8-bit KV and MTP off. The expert-cache budget, prefetch, prefill chunk and sampling defaults come from the install's own ModelProfile row, clamped to this machine's RAM, so the launcher overrides a measured optimum only when --ram is passed. A model served on the CPU backend takes none of the pinned flags: that backend has no prompt cache and no quantized KV. That is the backend, not the install -- a dense Qwen 3.5 model served on the GPU gets the GPU profile, and the same install served as <id>@cpu gets the CPU one.

Warning

The launcher stops an TinyTitanServer already listening on its port so it can bind, and it never touches a process on that port that is not TinyTitanServer. Start the server by hand when another session must stay up. The launcher asks for the port and uses 8080 if you press Enter; --port or TINYTITAN_PORT answers it without the question. On any port other than the default the launcher prints the command that re-points the DeepSeek Harness route.

Connect a client

Use this local provider information:

Setting	Value
Base URL	http://127.0.0.1:8083/v1
API key	Any non-empty value when the client requires one
Ornith base model	ornith-1.5-35b-a3b_8-Bit (or _4-Bit)
Ornith chat-only alias	ornith-1.5-35b-a3b_8-Bit-fast
Qwen 3.6 base model, when served manually	qwen3.6-35b-a3b_8-Bit (or _4-Bit)
Qwen 3.6 chat-only alias	qwen3.6-35b-a3b_8-Bit-fast
Qwen3.8-Flash-Next, when served manually	qwen3.8-flash-next_4-Bit

Codex configuration:

model = "ornith-1.5-35b-a3b_8-Bit"
model_provider = "tinytitan"

[model_providers.tinytitan]
name = "TinyTitan"
base_url = "http://127.0.0.1:8083/v1"
wire_api = "responses"

Set any API key required by the client, for example OPENAI_API_KEY=local. The CLI launcher writes isolated Codex and Qwen Code configurations automatically; it does not replace their normal config homes.

DeepSeek Harness is not a launcher entry: it reaches the server through its own llm-pi-ai provider route. Do not hand-write that route: this checkout generates it from the installs it has —

tools/dsh_route.sh                 # print the block
tools/dsh_route.sh --write         # replace the llm-pi-ai section in ~/.dsh/settings.yaml
tools/dsh_route.sh --models qwen38 --reasoning off --write

It reads the server's own catalog (or TINYTITAN_CATALOG_JSON), so the picker follows models/ — served ids, each template's thinking levels, and the three switches that are easy to get wrong by hand. Only installs really present are listed, one row per width.

plugins/dsh-tinytitan/ does the same at DSH boot, watches models/ while the harness runs so an install or a deletion reaches the picker without a restart, reads the folder directly when no server has been built yet, and additionally mounts a compaction backend that forces thinking off for the auxiliary calls. What it writes, for reference, looks like this (port 8083 to match this page; 8080 by the launcher's default):

llm-pi-ai:
  providers:
    tinytitan:
      displayName: TinyTitan
      api: openai-completions
      baseURL: http://127.0.0.1:8083/v1
      headers:
        authorization: Bearer tinytitan-local   # pi-ai refuses a keyless route; the server has no auth
      reasoning: medium                      # level for calls that name none (compaction, session titles)
      streamIdleTimeoutMs: 3600000           # a cold local prefill outlives pi-ai's five-minute default
      models:
        - id: qwen3.8-flash-next_4-Bit       # exactly what GET /v1/models lists
          name: Qwen 3.8 Flash Next 125B-A6B (4-bit)   # what the picker shows; the width is part of it
          contextWindow: 262144              # the launcher's "context=" value
          maxTokens: 32768
          reasoningEfforts:
            off:
            low: low
            medium: medium
            xhigh: xhigh                     # what the Qwen3.8 template renders (default xhigh)
          compat:
            thinkingFormat: chat-template    # the only place TinyTitan reads enable_thinking; pi-ai's `qwen` format sends it top-level
            chatTemplateKwargs:
              enable_thinking: { $var: thinking.enabled }
              reasoning_effort: { $var: thinking.effort }
            maxTokensField: max_tokens
            supportsUsageInStreaming: true

supportsDeveloperRole: false is no longer needed — the server serves developer as system — but a route may keep it to send the identical wire.

Verified 2026-09-14 against qwen3.8-flash-next_4-Bit started with tools/server_launcher.sh --client server --model qwen38 --bits 4: thinking off and on per request against a server loaded either way, thoughts returned as reasoning, and completion_tokens_details.reasoning_tokens in the usage. KAT is the binary-thinking family, so its route declares off: plus one level mapped to on.

Base model or fast alias

Use the selected model's base ID for coding agents and tool loops. Append -fast for direct questions.

The fast alias uses the same weights but removes coding-CLI system prompts, tool definitions, tool history, and known reminder blocks before prefill. This can reduce first-response wall time substantially, but it also removes the information required for tool calls. It is chat-only for that request.

Prompt reuse

Multi-prefix reuse is enabled by default. Send the complete conversation each turn. A compatible continuation reports reused tokens in:

usage.prompt_tokens_details.cached_tokens

The default cache keeps up to four prefixes in RAM with a 256 MiB snapshot budget. To persist compatible prefixes across restarts:

.build/release/TinyTitanServer \
  --model models/ornith-1.5_35B_A3B_8Bit \
  --port 8083 \
  --prompt-cache-disk "$HOME/Library/Caches/TinyTitan/prompt-cache" \
  --prompt-cache-disk-mib 8192

The disk cache can contain conversation text and source code. Keep it private, local, and unsynchronized. Prompt reuse improves prefill and time to first token; it does not increase decode tokens per second.

Model residency

By default the server loads at startup and remains resident. For a shared Mac:

• --lazy-load loads on the first inference request.
• --idle-unload-seconds N unloads after an idle period and implies lazy load.
• POST /v1/models/unload unloads on demand when dynamic residency is enabled.

Pair idle unloading with --prompt-cache-disk; unloading discards the in-memory prompt cache.

Tool calls

TinyTitan returns function-call requests but never executes them. The client must apply its normal permission policy, execute approved tools, append each result with the matching tool_call_id, and resend the complete history. Only function tools and automatic or disabled tool choice are supported.

Next steps

• Essential tuning: Runtime Controls
• Common connection problems: FAQ
• Reproducible measurements: Benchmarking Guide