🐪 Camelid

A Rust-native local LLM inference engine — GGUF in, OpenAI-style API out, every claim backed by reproducible evidence.

Camelid loads GGUF models directly, serves them over a local OpenAI-style API, and gates every optimized path on token-for-token parity with a reference implementation. It is not a wrapper around Ollama or llama.cpp — the tokenizer, GGUF loader, CPU kernels, and the Metal (Apple Silicon) and CUDA (NVIDIA) GPU paths are all implemented in this repository, shipping as a single static Rust binary with no Python.

_{The local web frontend — a dark, collapsed-rail chat surface that unlocks chat only for model rows the compatibility contract recognizes.}

---

Install

Two ways to run Camelid — both use the same engine and the same models. Pick what fits:

	🪟 Camelid Desktop	⚙️ Camelid engine
What it is	A native Windows app	The prebuilt camelid binary
Best for	Just chatting on your own PC — the easy button	Sharing on a network, the API, scripting
How you chat	A native window (no browser, no terminal)	In your browser, or a server others connect to
Install	Double-click the signed installer	Unzip and run camelid.exe
Runs on	Windows	Windows · macOS · Linux

The desktop app simply wraps the same engine in a native window — same models, same support gate, same GPU acceleration. If you just want to chat, get the desktop app. If you want to share it or use the API, get the engine.

🪟 Camelid Desktop (Windows) — easiest

The simplest way to run a model locally: a native app, no browser tab, no command line.

_{Camelid Desktop on Windows — the same chat surface in a native window, with the supported Llama 3.2 3B row loaded and ready.}

1. Download the Camelid Desktop installer (Camelid.Desktop_<version>_x64-setup.exe) from the latest release.
2. Double-click it and follow the prompts. It's code-signed (verified publisher); if Windows SmartScreen warns on a fresh download, click More info → Run anyway.
3. Launch Camelid Desktop from the Start menu. It starts the engine for you and opens the chat window — pick a model to download, and chat.

Installs just for you under %LOCALAPPDATA%\Camelid Desktop — no admin rights needed. GPU acceleration works on any NVIDIA card with only the normal driver (the CUDA runtime is bundled); no GPU, it runs on the CPU.

⚙️ Camelid engine — run it in a browser, or serve it to share

The prebuilt binary, with the web UI baked in. Run it to chat in your browser, or serve it so other people or apps can connect over an OpenAI-style API — on Windows, macOS, or Linux. Get it from the latest release.

Windows (x86_64):

1. Download camelid-windows-x64.zip and right-click → Extract All… to a folder (your Desktop is fine).
2. Run .\camelid.exe serve in a terminal (or double-click camelid.exe).

The chat UI opens automatically at http://127.0.0.1:8181 in your default browser. The binary is Authenticode code-signed, and GPU acceleration works out of the box on any NVIDIA card (normal driver only — the CUDA runtime is bundled). No Python, Node, Docker, or CUDA Toolkit to install.

macOS (Apple Silicon) / Linux (x86_64):

# macOS (Apple Silicon)
curl -L https://github.com/timtoole02/Camelid/releases/latest/download/camelid-macos-arm64.tar.gz | tar -xz
cd camelid-macos-arm64
xattr -d com.apple.quarantine ./camelid 2>/dev/null || true   # allow the unsigned binary to run

# Linux (x86_64): same, with camelid-linux-x86_64.tar.gz

Then: download a model and chat (any OS):

./camelid pull llama32_3b      # download a supported model into ./models
./camelid serve --model models/Llama-3.2-3B-Instruct-Q8_0.gguf

serve runs one static binary serving the OpenAI-style API and the web UI on the same port. Sharing on a network? camelid serve --addr 0.0.0.0:8181 lets anyone on your LAN open the same chat UI and API. Prefer to build from source? See Build from source.

---

Why Camelid is different

Most local runtimes optimize for breadth — "point it at any GGUF." Camelid optimizes for trust, and treats the boundary as the feature:

• Every claim is backed by a re-runnable receipt. Support is per exact model row — a specific GGUF at a specific quant — and an optimized path ships only after it matches a reference token-for-token. No "same family, probably fine."
• It fails closed, on purpose. Point it at an unsupported model and you get a typed error, not a silent wrong answer. The honest boundary is the product.
• One Rust binary, no Python. Tokenizer, GGUF loader, CPU kernels, and the Metal and CUDA GPU paths all live in this repo and ship as a single static binary — serve even embeds the web UI.
• Numbers come with logs or they don't ship. Every published benchmark links to a committed bundle with raw logs, exact commands, and versions. No raw log, no claim.

---

More ways to run

You already have the web chat UI from serve (see Install above). Camelid also runs in your terminal and as a sandboxed agent — covered just below.

Want a single command that proves the whole path end to end? scripts/smoke.sh pulls TinyLlama, serves it, does one real chat round-trip, and asserts on the reply — no mocks. See Quickstart.

Run in the terminal

Prefer the keyboard? camelid chat is a full-screen terminal app — Markdown-rendered replies that stream in live, a scrollable chat pane, a settings sidebar with a context gauge, a / command palette, and instant switching between models already loaded in the server. It attaches to a running camelid serve or spawns one for you. (Over a pipe, SSH without a TTY, or with --plain, it falls back to a scrollback-friendly line REPL.)

./camelid pull tinyllama        # the baseline supported row (or any pull alias)
./camelid chat                  # full-screen TUI; opens the model browser, or:
./camelid chat --model models/tinyllama-1.1b-chat-v1.0.Q8_0.gguf

Type / to open the command palette and browse everything (filter as you type, ↑↓ to pick, Tab/Enter to run). Highlights: /models browses loaded + downloadable models, /switch flips instantly between models already loaded in the server (no reload), /set <temperature|top_p|top_k|max_tokens|seed|stream> <value> tunes sampling live, /system sets a prompt, /save//load persist a session, /copy yanks the last reply to the clipboard, /theme restyles, /retry regenerates. Tab toggles the sidebar, PgUp/PgDn and the wheel scroll, Ctrl-C stops a stream, Ctrl-D quits (F1 for the full key/command list). The model browser is built from the live support ledger (/api/capabilities) — it lists only supported rows and shows which are already downloaded. Pointing --model at a GGUF whose architecture Camelid doesn't support is refused with the same typed error the rest of the engine uses — the terminal is not a backdoor around the support contract. Gemma 4 12B/26B remain two-Mac distributed only and are not single-node chat rows.

Agent mode (preview)

camelid chat --agent --model <gguf> runs a sandboxed tool-calling loop: the model reads/writes/searches files, runs shell commands, and (opt-in) fetches URLs, observing each result and iterating toward your goal — every write/exec/network action behind an approval prompt (y once · a this tool for the session · n deny · q abort). File tools are confined to a canonical workspace root (--workdir, default the current directory); path escapes (.., outside-absolute, escaping symlinks) are refused in code, not just discouraged. Tool results are treated as untrusted data — an instruction hidden in a file or web page can never make the agent escalate or run a prohibited action. The network tool is off unless --allow-net; --auto-approve exists for power users but warns loudly and still enforces the sandbox.

Requires a tool-capable supported row. Agent mode is gated to models the ledger marks tool_capable, promoted only with a real tool-call round-trip as evidence — the same bar as the support gate. The engine renders tool definitions through each model's own chat template (canonical flat-function form, matching llama.cpp/vLLM — see TOOLCALL_DIAG.md) and the loop parses the tool-call output back out; that plumbing ships and is tested.

Promotion is decided by the camelid agent-eval --model <gguf> harness, which runs a fixed tool-use battery against a fixture and reports one of three outcomes with a receipt artifact: PASS (clean round-trip — eligible for promotion), FAIL (loaded but the model can't produce usable tool calls), or INCONCLUSIVE (didn't load in budget — a contended box, not a capability failure; re-run on a quiet host). A row's tool_capable flag is flipped true only after a PASS receipt — never a lucky run.

Llama 3.2 3B Instruct Q8_0 is the first promoted row — it earned a PASS receipt (qa/agent-eval/): with the corrected render it emits well-formed tool calls, reads the fixture, and answers correctly. So camelid chat --agent --model models/Llama-3.2-3B-Instruct-Q8_0.gguf runs the live loop. (The 1B is too weak — it FAILs the harness with malformed args even with the correct render — so it stays gated, as does any row without a PASS receipt.) The capability moves only on harness evidence, never a claim.

Native Windows desktop app (add-on)

Prefer a desktop window over a browser tab? Camelid Desktop is an additive native Windows app (Tauri v2 + WebView2) that embeds the same camelid engine — it spawns camelid serve as a loopback sidecar and hosts the existing web UI in a native window. It inherits the identical support contract and the same runtime-ready + exact-supported-row chat gate (it talks to the same /api/capabilities); it makes no broader claims about supported models or performance, and any tokens/sec readout is sourced from real generation events. The web path remains canonical. See camelid-desktop/README.md.

---

Which model should I try first?

Every row below is a supported exact row with committed evidence; the caveat column is the real support envelope from STATUS.md, not marketing. The three rows in camelid pull are the frictionless path — pick one and you're chatting in two commands (or run scripts/smoke.sh for the zero-decision path).

If you want…	Try this row	One command	First-run reality (from STATUS.md)
The fastest "does it work" check	TinyLlama 1.1B Chat Q8_0	camelid pull tinyllama	The baseline gate — ~1.2 GB, single-node, runs anywhere. This is exactly what scripts/smoke.sh exercises.
A solid single-node default	Llama 3.2 3B Instruct Q8_0	camelid pull llama32_3b	Exact-row smoke + API/WebUI, single-node Apple Silicon or CPU. Verified context is bounded to 512/1024/2048 — longer contexts aren't a support claim yet.
A small Gemma 4	Gemma 4 E4B-It Q8_0	camelid pull gemma4_e4b	5/5 greedy parity vs the pinned llama.cpp oracle on the CPU runtime and the Metal GPU-resident runtime. Windows NVIDIA CUDA decode is experimental (first-token argmax matches the CPU oracle; not token-for-token gated). Multimodal input fails closed by design.

Also supported — bring the official Q8_0 GGUF and point serve at it (these exact rows aren't in camelid pull yet):

• Most capable on a 16 GB Mac — Mistral 7B Instruct v0.3 Q8_0. Exact-row smoke with bounded context 512→8192 and GPU-vs-CPU greedy parity; the 7B parity receipt re-verifies on a 16 GB host.
• Kick the tires on Qwen — Qwen3 1.7B Q8_0 (Qwen/Qwen3-1.7B-GGUF). ChatML with token-and-text parity at 1/5/50 tokens plus API smoke (thinking-disabled is the parity-locked mode). Runs on the GPU-resident decode + single-shot prefill path (per-head QK-norm applied in-kernel), validated token-and-text-identical to llama.cpp at a 15,373-token single-shot prefill context (ceilings: 16,384 single-shot prefill / 40,960 KV). Thinking mode is available opt-in (camelid_enable_thinking:true): the model emits its own <think>…</think> reasoning, token-identical to llama.cpp for the leading trace (26–205-token envelope) before the documented f32 frontier.

Mostly a two-machine setup: Gemma 4 12B-It is in active validation only through the two-Mac distributed serve lane (not promoted to full support) — single-node on a 16 GB CPU host is memory-bound and unsupported. Treat it as a deliberate two-machine setup (docs/gemma4-two-mac-cluster.md). The 26B-A4B MoE now also runs single-node on a 6 GB Windows NVIDIA card via the opt-in SSER (self-specializing expert residency) expert-offload lane — experimental (bit-exact greedy vs the CPU oracle in-tree; no committed evidence bundle yet).

Anything not in Supported models is refused or clearly labeled: unimplemented architectures fail closed with a typed error, and a file whose architecture is implemented but whose exact row isn't supported runs only in a clearly-marked experimental lane — every reply flagged as unverified, with no parity claim. That's the contract, not a limitation to work around.

---

At a glance

🦀 Rust-native	Tokenizer, GGUF loader, CPU kernels, and the Metal and CUDA GPU paths live in this repo. One static binary, no Python.
📦 Direct GGUF	Point it at a .gguf file — no conversion or import step.
🔌 OpenAI-style API	/v1/chat/completions and /v1/completions with SSE streaming, served locally.
✅ Correctness-first	Optimized paths ship only after token-for-token parity with a reference; unsupported configs fail closed with typed errors.
🧾 Proof-carrying	Any request can emit a sealed parity receipt — exact GGUF (SHA-256), exact input, exact tokens — independently re-verifiable against llama.cpp on your own machine, including 7B receipts on a 16 GB Mac.
📊 Evidence-gated	Every published number comes from a committed bundle with raw logs, commands, and versions. No raw log, no claim.
⚡ Apple Silicon path	A Metal-resident pipeline (GPU prefill, GPU decode with on-GPU greedy sampling) measured head-to-head against llama.cpp and MLX-LM — wins, ties, and losses all stated.
🖥️ NVIDIA CUDA path (Windows)	A from-scratch NVRTC GPU-resident decode + prefill engine — no vendored llama.cpp — token-parity-validated per exact row, with fully-GPU-resident 9B rows on a 6 GB card.
🚀 Fast model loading	On Apple Silicon the server maps Q8_0 weights for the GPU to read in place instead of reading and copying them, so reloads are quick and peak memory stays lower.

---

Supported models

Support is per exact model row (a specific GGUF at a specific quantization), each backed by committed evidence. Anything not listed either fails closed (unimplemented architecture) or runs only in the clearly-marked experimental lane (implemented architecture, no supported row, no parity claim).

Model row	Quant	Serve lane	Evidence
TinyLlama 1.1B Chat	Q8_0	single-node	Current verified gate
Llama 3.2 1B Instruct	Q8_0	single-node	Exact-row + bounded context 512→8192
Llama 3.2 3B Instruct	Q8_0	single-node	Exact-row smoke + API/WebUI + bounded context
Llama 3 8B Instruct	Q8_0	single-node	Exact-row + bounded context 512→2048
Mistral 7B Instruct v0.3	Q8_0	single-node	Exact-row smoke + bounded context 512→8192 + GPU/CPU parity
Qwen3 1.7B	Q8_0	single-node (CPU + CUDA)	Exact-row ChatML (thinking-disabled) — token+text parity at 1/5/50 tokens + API smoke; macOS GPU-resident decode+prefill validated to a 15,373-token context (vs llama.cpp); Windows CUDA GPU-resident decode+prefill parity (== cpu_reference/llama.cpp at 1/5/50, RTX 3060 Laptop); thinking mode opt-in (leading-trace parity)
Qwen3 0.6B	Q8_0	single-node (CPU + CUDA)	Exact-row ChatML (thinking-disabled) — token+text parity at 1/5/50 tokens (explicit head_dim path); Windows CUDA GPU-resident parity (== cpu_reference/llama.cpp); thinking mode opt-in (leading-trace parity, 6–126-token envelope)
Qwen3 4B	Q8_0	single-node (CPU + CUDA)	Exact-row ChatML (thinking-disabled) — token+text parity at 1/5/50 on confident prompts (explicit head_dim); one probe is a documented first-token near-tie; Windows CUDA GPU-resident parity (== cpu_reference/llama.cpp); thinking mode opt-in (leading-trace parity, 35–235-token envelope)
Qwen3 8B	Q8_0	single-node (CPU + CUDA)	Exact-row ChatML (thinking-disabled) — token+text parity at 1/5/50 tokens (untied embeddings); on the macOS GPU-resident decode+prefill path; Windows CUDA via VRAM+host-RAM offload (16/36 layers resident on a 6 GB card, parity == cpu_reference/llama.cpp at 1/5/50); thinking mode opt-in (template-shape byte parity + host-bounded leading-trace)
Ornith 1.0 9B (qwen35 hybrid state-space/SSM)	Q8_0 (CPU) · Q4_K_M, Q3_K_M (Windows CUDA, fully GPU-resident on 6 GB)	single-node (CPU + CUDA)	From-scratch qwen35 engine (gated DeltaNet linear attention + sparse full attention, 24+8 layers). Greedy token parity vs pinned llama.cpp acd79d6: 4/4 committed at n=20 (Q8_0 CPU) + five-prompt n=64 extension with every flip probed and attributed to sub-0.33-nat near-ties (the oracle's own backends flip at the same positions); byte-exact BPE tokenizer gate (45 fixtures × 2 modes incl. NFD/Devanagari/ChatML adversarial); tool_capable on Q8_0 and Q4_K_M (a committed agent-eval battery PASS on each — Q3_K_M has no agent-eval receipt yet and is not tool-capable); reasoning_content + qwen3_xml → tool_calls serving incl. SSE streaming with include_usage; Q3_K_M runs 16K context fully resident (4.7 GB peak, ≥1.3 GB headroom) at ~15 tok/s, Q4_K_M @8K at ~19 tok/s, greedy == CPU oracle. Receipts under qa/ornith/
Gemma 4 E2B-It	Q8_0	single-node (CPU + Metal)	5/5 greedy parity (CPU + Metal) vs pinned llama.cpp 5d56eff
Gemma 4 E4B-It	Q8_0	single-node (CPU + Metal); CUDA experimental	5/5 greedy parity (CPU + Metal) vs the pinned reference. Windows NVIDIA CUDA decode is experimental — first-token argmax matches the CPU oracle (in-tree gate, RTX 3060 Laptop); not token-for-token gated, no committed evidence bundle yet
Gemma 4 12B-It	Q8_0	two-Mac distributed	Active validation (two-Mac distributed ONLY, not promoted): distributed output == single-node + 3/5 full-budget vs the pinned reference + serve/WebUI smoke
Gemma 4 26B-A4B-It QAT	Q4_0 (128-expert MoE)	two-Mac distributed; single-node Windows CUDA (experimental)	Active validation (two-Mac distributed): 2/5 full-budget + 3/5 frontiers vs the pinned reference + serve/WebUI smoke. Windows NVIDIA CUDA single-node GPU-resident now runs this 26B MoE on a 6 GB card via the opt-in SSER expert-offload lane (attention + dense on the GPU; the 128 experts stay host-resident with an adaptive VRAM expert cache): experimental — greedy tokens bit-identical to the CPU oracle on content (in-tree, RTX 3060 Laptop; within the f16-KV floor), ~6–7 tok/s, opt-in via CAMELID_GEMMA4_CUDA=1 + CAMELID_SSER_CACHE=1; no committed evidence bundle yet
DiffusionGemma 26B-A4B-It	Q4_K_M	single-node (CPU; Windows CUDA)	Experimental — bit-exact through the full chat path (Phases 0–6) vs the pinned reference (Apple Silicon); verified running on Windows x86_64 (MSVC) with model receipts (determinism pairs + a full 2-block 512-token multi-canvas answer), pure-Rust (no C/C++). Run via camelid diffusion-gemma-chat (--max-steps N bounds the denoise). GPU (CUDA, on by default on Windows): the self-conditioning matmul runs on the GPU (2.7× wall, documented non-bit-exact; CAMELID_DG_CUDA_SC=0 opts out) and the MoE experts run from a VRAM-resident expert pool that is bit-exact — an experts-on-GPU run is byte-identical to the CPU-pure oracle (fits a 6 GB card: 4.6 GiB peak)

Fails closed (by design): Mixtral-8x7B v0.1 (validation-in-progress, one-token runtime only); other Qwen3 sizes (14B/32B), base variants, Qwen3-MoE (A3B), and full-trace Qwen3 thinking-mode token-parity (thinking is available opt-in with leading-trace parity); Gemma 4 26B-A4B Q8_0 (26.9 GB) and 31B (over the 2×16 GB envelope); Gemma 4 MTP/drafter rows; DiffusionGemma 26B-A4B on the autoregressive engine (a discrete block-diffusion model cannot run an AR forward — the AR engine fails closed and redirects to the dedicated diffusion-gemma-chat lane, which is supported; see below); multimodal input; and all other quantizations in v0.1.

Experimental lanes

Model row	Quant	Status	Evidence
DiffusionGemma 26B-A4B-It	Q4_K_M	Supported (experimental) via the dedicated diffusion lane (camelid diffusion-gemma-chat, --max-steps N). Pure-Rust on macOS / Linux / Windows (the expert-argsort C++ shim was removed). CPU multi-step is slow (the self-conditioning matmul dominates). GPU (Windows CUDA, evidence-backed): the SC matmul offloads for 2.7× wall (non-bit-exact f32 accumulation, CAMELID_DG_CUDA_SC=0 opts out; CAMELID_DG_CUDA=0 for CPU-pure) and the MoE expert pool holds ~2.9 GiB of experts VRAM-resident with bit-exact GEMV kernels — byte-identical output vs the CPU-pure run (recon §8e). The autoregressive engine fails closed and redirects here by design.	End-to-end bit-exact vs the pinned llama.cpp diffusion reference at zero tolerance, recon: Phase 0.5 lazy dequant (5 quant formats) + Phase 1 tokenizer (12/12, 100% token-id match) + Phase 2 encoder checkpoints (242/242, 510/510 expert selections) + Phase 3 single denoise step (all 67,108,864 canvas logits + host-RNG streams + every EB step-0 output) + Phase 4 full EB denoise loop (S=48, live self-conditioning, 268M logits) + Phase 5 multi-canvas block-autoregressive loop (2 blocks, 512-token response byte-identical to the reference) + Phase 6 chat wrapper (render+tokenize and detokenize parity vs the reference chat path). CPU-pure pinned configuration
Gemma 4 E4B-It	Q4_0 (mixed QAT)	Experimental — Windows NVIDIA CUDA, off by default (CAMELID_GEMMA4_CUDA). The mixed-quant export (Q4_0 projections, Q4_1 ffn_down, Q4_K tied head, Q5_K per_layer_token_embd, BF16 proj) runs GPU-resident in ~2.5 GB — fits a 6 GB card with headroom — at ~18 tok/s decode. Not a supported row: the first-token argmax matches the CPU oracle and every projection GEMV is bit-exact, but the fp-reassociated attention/Per-Layer-Embedding (PLE)/norm reductions flip later near-ties on the coarse Q4 logits, so it is argmax-stable, not token-for-token greedy parity (unlike the supported E4B Q8_0 CUDA row).	Per-kernel bit-exact unit tests (q4_0 / q4_1 / q4_k GEMV, 96/96 rows each) + an in-tree first-token parity gate vs the CPU oracle loading the same file (RTX 3060 Laptop, 6 GB)
Llama 3.2 3B Instruct	Q4_K_M, Q5_K_M	GPU-resident raw-decode parity, held out of the runtime support gate — /api/capabilities does not advertise these yet (recognition pending API/WebUI/serve smoke). GPU-resident CUDA decode drives the q4k_gemv/q5k_gemv + q6k_gemv kernels on the mixed K-quant tensors, direct against llama.cpp (no camelid CPU K-quant decode path). No speed, neighboring-quant, or family claim.	Q5_K_M: token-and-text-identical to pinned llama.cpp acd79d603 at 1/5/50 tokens (all_pass=true). Q4_K_M: token-and-text-identical on 5/8 confident probes at 1/5/50 (incl. code completion + a long-context continuation to depth 50); 3 open-ended probes diverge at benign greedy f32 near-ties. Bundles under qa/evidence-bundles/llama-3.2-3b-q{4,5}_k_m-windows-cuda-resident-parity-*/

Windows bring-up, perf & GPU (2026-06/07). The diffusion lane builds and runs verified on Windows x86_64 (MSVC) with zero C/C++: the expert-argsort std::sort shim was ported to pure Rust, leaving only macOS-only Apple framework bindings (vDSP / __sincosf_stret) elsewhere. The portable sort breaks an exact expert-probability tie by lower index rather than reproducing the reference's libc++ introsort tie-order, so re-validate the Apple-Silicon encoder/decode parity gates if exact-tie ordering matters. GPU offload now has committed evidence (recon §8e; RTX 3060 Laptop 6 GB): CPU-pure and GPU determinism run-pairs both byte-identical; the self-conditioning matmul on GPU cuts a 2-step leg from 551 s to 200 s (2.7×, the one non-bit-exact stage — f32 accumulation, per-stage opt-out CAMELID_DG_CUDA_SC=0); the MoE expert pool keeps ~2.9 GiB of expert weights VRAM-resident behind bit-exact Q4_K/Q8_0 GEMV kernels (unit-gated, and an experts-on-GPU run is byte-identical to the CPU-pure oracle) — a capacity lever that also lets the 16.8 GB model coexist with a 16 GB-RAM host; and a natural-stop 2-block multi-canvas answer (512 tokens, grown prefix 297 matching the Apple-sealed Phase 5 shape) completed end-to-end at 4.6 GiB VRAM peak. Windows claims are a this-host contract (determinism + bit-exact GPU stages vs the local CPU oracle), not a bitwise claim against the Apple-sealed artifacts (off-macOS sincos + tie-order notes in the recon).

Per-row detail and the exact evidence artifacts live in SUPPORT_MATRIX_v0.1.md and COMPATIBILITY.md.

---

Engine status

Capability	Status	Notes
GGUF loading	✅ Working	Direct load with metadata/tensor inspection (camelid inspect).
Q8_0 inference	✅ Working	The validated quantization; support is per exact row (see above).
Gemma 4 engine	✅ Working	From-scratch gemma4 engine — see Gemma 4 below.
OpenAI-style API	✅ Working	/v1/chat/completions, /v1/completions, /v1/models, plus capability/health routes.
Streaming chat	✅ Working	SSE streaming on the chat endpoint, including OpenAI stream_options.include_usage — opt in for a terminal usage chunk whose prompt_tokens/completion_tokens/total_tokens are identical to the non-streaming response.
Apple Silicon Metal path	✅ Working	GPU-resident prefill and decode, auto-selected when a Metal device is present; CPU fallback otherwise.
NVIDIA CUDA path (Windows)	✅ Working	GPU-resident decode + single-shot prefill (--features cuda), auto-selected when a CUDA device is present; token-parity-validated on the dense Qwen3 Q8_0 rows (RTX 3060 Laptop). The Gemma 4 E4B-It Q8_0 row also runs on this path but is experimental (first-token argmax matches the CPU oracle; not token-for-token gated, no committed bundle yet). Models that exceed VRAM use automatic VRAM+host-RAM layer offload; CPU fallback otherwise. Results are GPU/driver/CUDA-version specific.
Web frontend	✅ Working	Local React/Vite chat surface, embedded in the binary and served at the same address; unlocks chat only for recognized model rows.
Parity receipts	✅ Working	Opt-in sealed record of one request; camelid verify-receipt re-checks it against llama.cpp (incl. 7B on a 16 GB host).
Two-Mac distributed serve	✅ Working	Layer sharding over TCP for rows too large for one 16 GB host (Gemma 4 12B, 26B-A4B).
Other quantizations	⛔ Not supported	No supported row; implemented architectures may load in the experimental lane (unverified, no parity claim), everything else fails closed.
Ghost mode (layer streaming)	🧪 Experimental	ghost-run executes one block at a time for a strict memory ceiling; trades throughput for memory.

---

Ornith 1.0 9B (qwen35)

Camelid implements the qwen35 architecture (Qwen3.5 / Ornith-1.0-9B, a hybrid of gated-DeltaNet linear-attention (state-space, SSM) layers and sparse full attention — 24 SSM + 8 full-attention layers) from scratch: the gated delta-rule recurrence, causal conv1d, partial NEOX mRoPE, gated attention, and a hybrid cache (fixed-size recurrent state for DeltaNet layers, f16 KV for the full-attention layers — context length does not grow the SSM state). The reference pin is llama.cpp acd79d6 with CUDA (REFERENCE_PIN_QWEN35.md).

The model is a reasoning model with tool calling: turns open with <think>…</think> (split into reasoning_content, message and SSE deltas) and tools are called in its custom qwen3_xml format, lifted into OpenAI tool_calls. The Q8_0 and Q4_K_M rows are tool_capable, each with a committed agent-eval PASS receipt (Q8_0: three camelid.agent_eval/v1 PASS receipts on the runnable lane; Q4_K_M: a full read/list/write agent battery PASS on the GPU-resident file). The Q3_K_M row carries no agent-eval receipt yet and is not tool-capable.

On the recorded 6 GB Windows CUDA card (RTX 3060 Laptop), Q4_K_M and Q3_K_M run fully GPU-resident — sparse KV keeps 16K context inside 6 GB on Q3_K_M (4.7 GB peak) — with a device-side decode loop (on-GPU embedding gather + resident rope tables; the host syncs once per token chunk). Decode is ~19 tok/s (Q4_K_M @8K) / ~15 tok/s (Q3_K_M @16K), greedy-token-identical to the CPU oracle lane. The Q3_K_M and IQ-family quants, an imatrix calibrated on an agentic-coding corpus, and a quality × residency table are documented under qa/ornith/constrained-vram/; a speculative split-precision lane was evaluated and closed with a measured NO-GO receipt (acceptance 0.87 passed, the speed bar did not — see RECEIPT_ITEM5_acceptance_economics.json).

Gemma 4

Camelid implements Gemma 4 from scratch in the gemma4 engine: per-layer-type sliding/global attention (the GGUF sliding_window_pattern is authoritative; E2B is 4:1), per-layer FFN widths and KV-head counts, QK-norm, dual-θ RoPE, GeGLU, Per-Layer-Embeddings, cross-layer KV sharing, and the <|turn>/<turn|> chat markers with thinking-channel suppression. Multimodal input fails closed with a typed error.

E2B-It & E4B-It (Q8_0, single-node). Five-prompt greedy parity against the pinned llama.cpp oracle on both the CPU and the Metal GPU-resident runtime. A reproducible bounded-context harness (512 / 1024 / 2048 / 4096 / 8192) and pinned-comparator oracles are committed under qa/gemma4/, but bounded context is not a promoted support claim for these rows — the committed evidence bundles record exact-row text-token generation within the basic_v1 prompt-pack envelope only ("no bounded-context promotion"). The chat template is locked byte- and token-exact (qa/gemma4/template_shapes_v1.json, both thinking modes). A Metal GPU-resident decode path (camelid gemma4-generate-gpu) runs the full E4B forward on the GPU at the memory-bandwidth wall. The QAT row (gemma-4-E4B_q4_0-it, Q4_0 layers + Q6_K tied head) runs on the same GPU-resident path — the Q4_0 projections decode on the GPU (parity-gated wire GEMVs) and the Q6_K tied head runs on the CPU; on an M4 it is token-for-token identical to the CPU runtime and ~25 % faster warm (15.2 vs 12.2 tok/s). The per-block GPU↔CPU parity is gated in CI; the end-to-end GPU==CPU check runs locally (no GPU model in CI). See docs/performance/gemma4-qat-gpu-2026-06-11.md. The committed CPU QAT parity (E4B QAT basic_v1: 3/5 full-budget + 2 probe-verified frontiers) is unchanged.

E4B-It on Windows (CPU and NVIDIA GPU). Gemma 4 E4B-It Q8_0 runs on Windows on the CPU and now also on the NVIDIA CUDA GPU. The CUDA lane is a from-scratch GPU-resident decode path for the full E4B forward — resident layer weights, a captured decode graph, the tied head, sliding/global attention, dual-θ RoPE, QK-norm, GeGLU, and Per-Layer-Embedding injection all on-device — wired into serve behind --features cuda and CAMELID_GEMMA4_CUDA, and auto-engaged when a CUDA device is present. Its first-token argmax matches the CPU Gemma4Runtime oracle, asserted by the in-tree gate (gemma4_cuda_matches_cpu_greedy), which is #[ignore] and checks first-token argmax stability — later-token divergence is permitted on this lane, so it is not token-for-token greedy parity. As with the other CUDA rows, results are GPU/driver/CUDA-version specific, so the lane stays experimental beyond the recorded GPU (RTX 3060 Laptop, 6 GB) and the CPU path remains the correctness reference; there is no committed CUDA evidence bundle for this row yet.

12B-It (Q8_0) & 26B-A4B-It QAT (Q4_0, MoE) — two-Mac distributed. These rows are too large for a single 16 GB host, so the lane under active validation is distributed layer sharding over TCP (not promoted to single-node support): gemma4-master/gemma4-worker split one row across two machines with a versioned handshake and per-packet checksums, and distributed greedy output is asserted token-identical to single-node (tests/gemma4_distributed_parity.rs). The 26B row is a 128-expert MoE (Q4_0 experts + Q6_K tied head) with the dense shared-expert + sparse top-8 branch implemented end to end.

Proven on two 16 GB M4 Mac minis, full basic_v1 pack vs the pinned reference:

Row	Distributed = single-node	vs. reference
12B-It Q8_0	5/5 token-identical	3/5 full-budget + recorded comparator frontiers
26B-A4B-It QAT Q4_0	identical (f32 wire)	2/5 full-budget token-identical + 3/5 probe-verified knife-edge frontiers

Both rows serve over HTTP through the same lane — set CAMELID_GEMMA4_SERVE=1 plus CAMELID_GEMMA4_WORKER/CAMELID_GEMMA4_SPLIT, and /v1/chat/completions (incl. SSE) and /v1/completions route through a persistent master shard with per-request worker sessions (wire protocol v1). The distributed serve/WebUI promotion smoke is green for both. Evidence bundles are under qa/evidence-bundles/; setup is in docs/gemma4-two-mac-cluster.md.

Scope guardrails: these are exact-row claims only — no Gemma-family-wide support, and no model-native/larger context beyond the checked packs.

---

Quickstart

Already have a binary from Install? Skip to "Get a model" below. To build from source instead — the web UI is compiled into the binary, so build the frontend first and it gets embedded (one binary, no separate Node process at runtime):

(cd frontend && npm ci && npm run build)   # bundles the web UI
cargo build --release                       # embeds it into the binary

Build from source on Windows (x86_64, MSVC)

Windows x86_64-pc-windows-msvc is a tracked platform (see COMPATIBILITY.md → Platform support). Most users should grab the prebuilt signed Windows download in Install above (GPU acceleration included); build from source only if you want to modify Camelid. Prerequisites: the MSVC toolchain (Visual Studio Build Tools with the C++ workload — not MinGW), Rust via rustup with the x86_64-pc-windows-msvc host, and Node.js for the embedded web UI. Then, in PowerShell:

cd frontend; npm ci; npm run build; cd ..   # bundles the web UI
cargo build --release                       # embeds it into the binary
.\target\release\camelid.exe pull tinyllama # the baseline supported row
.\target\release\camelid.exe serve --model models\tinyllama-1.1b-chat-v1.0.Q8_0.gguf

The server behaves exactly as on the other platforms (listens on 127.0.0.1:8181, same OpenAI-style API + web UI). The TinyLlama 1.1B Chat Q8_0 baseline gate is verified on Windows with the same parity evidence as macOS/Ubuntu.

GPU (NVIDIA/CUDA) on Windows. cargo build --release --features cuda adds a CUDA backend with a GPU-resident decode engine (weights uploaded once, single-shot GPU prefill, on-device greedy/temperature sampling) implemented from scratch in NVRTC kernels — no vendored llama.cpp. It auto-engages when a CUDA device is present. The dense Qwen3 Q8_0 rows (0.6B / 1.7B / 4B / 8B Instruct, thinking-disabled ChatML) are validated on it: GPU decode + single-shot prefill are token-AND-text-identical to the camelid cpu_reference (transitively llama.cpp 9632) at 1/5/50 generated tokens, greedy. Gemma 4 E4B-It Q8_0 also runs on this CUDA lane (enabled with CAMELID_GEMMA4_CUDA), greedy-parity with the CPU Gemma4Runtime oracle via the in-tree gate (gemma4_cuda_matches_cpu_greedy); it has no committed evidence bundle yet, so it stays experimental beyond the recorded GPU. /api/capabilities reports the live path (selected_backend=cuda_resident_q8_runtime, cuda_resident_active=true). Validated on an RTX 3060 Laptop (6 GB), driver 576.83, CUDA 12.9; 0.6B/1.7B/4B are fully VRAM-resident and 8B runs via automatic VRAM+host-RAM layer offload. Results are GPU/driver/CUDA-version specific (f32 reduction order is GPU-specific), so the lane stays experimental beyond the recorded GPU; the CPU path remains the default and correctness reference. See COMPATIBILITY.md → Windows CUDA and the qa/evidence-bundles/qwen3-*-windows-cuda-resident-parity-* bundles. Building the feature needs the CUDA Toolkit (12.x; libraries loaded at runtime); running it needs an NVIDIA GPU + driver.

Get a model. Camelid validates specific Q8_0 rows (most GGUFs on the web are other quantizations and fail closed), so pull fetches a known-good one into ./models:

./target/release/camelid pull              # list the supported models
./target/release/camelid pull llama32_3b   # download Llama 3.2 3B Instruct Q8_0

Serve it (pull prints the exact command to run; the model is in ./models):

./target/release/camelid serve \
  --model models/Llama-3.2-3B-Instruct-Q8_0.gguf \
  --threads 4

The server listens on 127.0.0.1:8181 and opens the chat UI in your browser automatically (pass --no-open to disable). The same address serves the OpenAI-style API. List the loaded model (its id comes from the GGUF metadata):

curl -s http://127.0.0.1:8181/v1/models

Chat (replace the model id with the one returned above; add "stream": true for SSE):

curl -s http://127.0.0.1:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Llama 3.2 3B Instruct",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64,
    "temperature": 0
  }'

The web frontend is served by the binary itself at the same address — no extra step. For hot-reloading frontend development, run the Vite dev server separately (it proxies to a running camelid serve):

cd frontend && npm ci && npm run dev

---

Evidence

Benchmark claims are listed only when raw logs or reproducible commands are committed. If there is no raw log, there is no benchmark claim.

Same-host snapshot on one Apple M4 (10-core GPU, 16 GB), Llama 3.2 3B Instruct Q8_0, greedy sampling, three same-session rounds with alternating runtime order (medians):

Lane	Camelid	llama.cpp (Metal)	MLX-LM (8-bit)
Prefill, 601-token prompt (tok/s)	587.3	543.7	577.9
Decode, short context (tok/s)	29.7	29.1	29.1

Reading boundary: a same-session result on one exact model row and one machine, with narrow margins — not a durable or general claim. Some lanes read below the comparators (decode at long context trails MLX-LM), and deeper prompt depths use single warm probes rather than protocol-grade rounds. Full methods, raw logs, per-round detail, and the lanes where Camelid loses are in BENCHMARKS.md and the bundles under qa/evidence-bundles/.

Correctness evidence (token-parity gates, per-row validation artifacts) is indexed in COMPATIBILITY.md and CORRECTNESS_v0.1.md.

Parity receipts

A parity receipt is a verifiable record of one request: the exact GGUF (by SHA-256), the exact input, and the exact tokens produced. Opt in with "camelid_receipt": true on /v1/chat/completions or /v1/completions, then check it on any machine:

camelid verify-receipt receipt.json --gguf path/to/exact-model.Q8_0.gguf

The verifier recomputes the receipt's digest, confirms your GGUF is the named file, replays the request through Camelid, and re-runs it through llama.cpp — in two isolated passes so each model loads within one model's memory footprint, which lets a 7B receipt verify on a 16 GB Mac. Receipts exist only for deterministic (greedy) runs; sampled runs are stamped reproducible: false. A receipt verifies a single request; it does not change the release ledger or promote any lane. Details in RECEIPTS.md.

To measure any local runtime — not only Camelid — by determinism, cross-runtime agreement, tokenizer parity, and provability on the same model bytes, see the conformance suite.

---

Under the hood

For the reader who wants the engineering, not the pitch — a few of the genuinely interesting artifacts:

• The token-major output.weight guardrail. TinyLlama had perfect tokenizer parity but wrong first-token logits until the final vocab projection was read as token-major rows. The fix, the rationale, and the regression guard are pinned in DECISIONS.md D0007.
• Reproduce any supported row's parity yourself. Each row's greedy parity is re-runnable with a committed harness against pinned llama.cpp — methodology and per-row reproduction steps in CORRECTNESS_v0.1.md.
• One four-row story across every surface. README, STATUS.md, /api/capabilities, and the UI are held to the same ledger by the readiness-gate inventory in VALIDATION_MATRIX.md — drift on any surface is treated as a bug.
• Sealed, portable parity receipts. Any greedy request can emit a SHA-256-anchored receipt that re-verifies against llama.cpp on a different machine (incl. a 7B receipt on a 16 GB Mac) — RECEIPTS.md.
• Engine internals. The from-scratch tokenizer, GGUF loader, CPU kernels, and Metal-resident pipeline are mapped in ARCHITECTURE.md.

---

Documentation

Doc	What's in it
SUPPORT_MATRIX_v0.1.md	Which exact model rows are supported, and with what evidence
COMPATIBILITY.md	The durable support contract
BENCHMARKS.md	Benchmark snapshots and claim rules
RECEIPTS.md	Verifiable single-request parity receipts
docs/CONFORMANCE.md	Measure any runtime by one ruler
STATUS.md	Current evidence snapshot and blockers
ARCHITECTURE.md	Implementation architecture
docs/gemma4-two-mac-cluster.md	Two-Mac distributed serve setup
RELEASE_NOTES_v0.1.md	v0.1 release notes
ROADMAP.md	Planned engineering sequence

Validation for code changes:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features

---

License

Camelid is licensed under the MIT License.

Camelid's tokenizer, reference compatibility layouts, and validation benchmarks are inspired by and checked against llama.cpp (Copyright © 2023–2026 The ggml authors, MIT License). Camelid maintains its own Rust-native codebase while crediting the reference work of the ggml ecosystem. Full third-party attributions are in THIRD_PARTY_NOTICES.md.

Star History