Coachwhip 0.5.2

Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers.

Faster on every Qwen3-Next and Qwen3.5/3.6 MoE model

• The attention block of a linear-attention layer is now one kernel per word. Three layers in four of these models use gated delta-net attention, and it was built from about forty small GPU operations per layer: the convolution taps, the activation, two norms, the widening, the delta rule over the memory, the output norm and the gate. They are now one Metal kernel per word; the two projections stay as their own matmul kernels. The arithmetic is the same: on the exact path the answers are byte-identical to 0.5.0.
• Measured by the tester on the release artifact, unpacked cold, fast mode, raw 300 tokens, three runs for the 35B: Qwen3.6-35B-A3B squeezed (bank 68) 13.0 → 15.2 tok/s (19.9, 19.7, 19.6 s); Qwen3-Coder-Next squeezed (bank 109) 11.8 → 13.8 tok/s. A warm route table (the engine learns which experts each layer wants, and the download starts cold) gave the builder 16.4 on the 35B; the tester's cold number is the one published. A 13,700-token read on the 35B: read at 64.9 tok/s, wrote at 15.7, 6.6 GB peak.
• COACHWHIP_ATTN_KERNEL=0 restores the old operation-by-operation path.

Where the time went, for the curious

On a 16 GB Mac the SSD stopped being the clock once experts were squeezed and stood in: a 35B word cost about 60 ms, only 10 of them in the expert layers. The rest was the attention side, and the delta step alone was 13 ms of it. The kernel takes most of that back.

Tested, and what it found

• Plain path, kernel on against off, same prompt, same model: the text is byte-identical on the 35B and on the coder. That was the release's one real risk and it held.
• gpt-oss-120b loads and answers on this build (a different attention; its numbers are 0.5.0's).
• The cache trap on the 35B compiles and runs; the two requirements it misses (an existing key not moved to most-recent on set; an O(n) eviction scan) are the model's habit, identical with the kernel off.

Not tested

• Macs other than an M4 with 16 GB.
• Thinking mode (--think) on the 35B through the new kernel; the 122B; prompts past 16K tokens.

Known

• --help still lists the supported families without gpt-oss; the text is stale, the support is there.

Coachwhip 0.5.0

Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers.

New model: gpt-oss-120b

• OpenAI's gpt-oss-120b on a 16 GB Mac (gpt-oss): 117 billion parameters, 4 of its 128 experts per word in each of 36 layers. It thinks before it answers, and the chat page shows the thinking.
• Squeezed once on your own Mac from the official MXFP4 file: coachwhip-prepare takes every expert matrix to Q2_K, rows padded to whole 256-weight blocks; everything else is copied as it is. 63.4 GB in, 42.6 GB out, in about 13 minutes on an M4; about 106 GB free is needed while both exist. Nothing is downloaded from us.
• Two modes. The model's own 4 experts per word, every one read for real: about 4.2 tok/s. Fast mode, --stand-ins: about 10 tok/s. Both measured on the same coding task (a cancellable async task pool in TypeScript, about 4,000 words of thinking and code, temperature 0): the first scored 4.8 of 5 on our 20 hidden checks, fast mode 5.0, 3.8 and 4.8 over three runs.
• Its own chat format. gpt-oss speaks OpenAI's harmony format; the chat page and the API speak it to the model, and the API returns only the final answer.

What to expect from it

• Everyday code, fast mode: working code at 8 to 10 tok/s. An arithmetic expression parser with a tokenizer, precedence climbing and error positions, asked for with traps in the requirements, came back at 8.3 tok/s and passed 10 of its 11 cases when run; the one it missed, unary minus binding tighter than ^, the original file with every expert read missed in the same way. On harder tasks it can get a detail like that wrong, in either mode. Read what it writes, and run it.
• Hard tasks are slow and weak. On a specification with several interacting rules it thinks for 3,000 to 5,000 words before writing, five to eight minutes at 10 tok/s, and a rate-limiter with asynchronous waiting scored 1.5 of 5 in fast mode and 1.1 with every expert read.
• --max-tokens counts the thinking. Start it with 8,000 as the setup shows; a lower cap can end a hard task inside the thinking, with no answer.
• The other models are unchanged: the 122B, the 80B and the 35B run from the same files with the same numbers as 0.4.1.

Faster

• Stand-ins, --stand-ins. When a light pick is not in the bank, Coachwhip uses the best expert that is, by the router's own scores, instead of reading one from the SSD. The first pick is always read for real, and the second whenever the router gives it at least 0.3 of the weight, and a substitute is used only when its router score is close to the real pick's. Fewer reads per word; the answers change, so check them for your use.
• Flash reads, --flash-reads. Experts are read straight from the SSD, past macOS's page cache, which has nothing to give back on a 16 GB Mac. The answers do not change.

Structure

• crates/engine/src/model_gptoss.rs: the gpt-oss family (attention sinks, a 128-word sliding window on every other layer, its clamped expert activation with biases). The expert maths is now a small hook per family, so the Qwen families are unchanged.
• Vendored candle-core with an MXFP4 kernel for Metal; its tests are left out.

Not tested

• gpt-oss beyond the chat's 16K tokens.
• Macs other than an M4 with 16 GB on macOS 26.
• --stand-ins on the Qwen models; it is meant for gpt-oss.

Coachwhip 0.4.1

The same Coachwhip as 0.4.0, now installed ready-built.

Install in under a minute

• The installer downloads a ready-built Coachwhip for Apple silicon from this release's page, checks it against its SHA-256 and unpacks it into ~/coachwhip. No Apple command line tools, no Rust, nothing to compile. It used to build from source, which took about 10 minutes and 3 GB.
• Same paths as before. The binaries sit where the build put them, ~/coachwhip/target/release/coachwhip and coachwhip-prepare, so every command in the README and the wiki works unchanged. coachwhip.sh starts the chat as before.
• Building from source is still there: run the installer with COACHWHIP_FROM_SOURCE=1. The installer also falls back to it if a release has no ready-built download.

Unchanged

• The engine, the models and the numbers are 0.4.0's. Only the version number moved; the measurements in docs/MODELS.md stand.

Not tested

• Macs other than an M4 with 16 GB on macOS 26.
• macOS versions older than the one the binary was built on.

Coachwhip 0.4.0

Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Method and every figure's conditions: docs/MODELS.md.

New model

• Qwen3.5-122B-A10B (qwen35moe), the biggest model Coachwhip runs on 16 GB, after a squeeze of 5 to 9 minutes on your own Mac: coachwhip-prepare takes the publisher's Q3_K_M file and writes a second file beside it with each expert's gate and up matrices at Q2_K (done on the GPU) and its down matrix at Q3_K; everything else is copied as it is. 45 GB written in 5 to 9 minutes on an M4; about 105 GB free is needed while the download and the squeezed file both exist. The squeeze is deterministic: the same download gave the same bytes on two runs on the Mac we tested. Nothing is downloaded from us.
• Two modes. Exact mode uses the model's own 8 experts per word. Fast mode, --experts 4 --parallel, asks the router for its top 4 and lets the GPU run ahead of the CPU: about twice the speed, and the answers can differ from the full model's. Fast mode is a choice, never the default; the README says so where the command is.
• Numbers on the squeezed file at bank 40, in docs/MODELS.md: the tester's run.

Faster

• The parallel path, --parallel. The GPU waits on a shared event for a layer's experts instead of the CPU re-queuing each layer; the CPU never waits for the GPU inside a word. Off by default; answers identical with it on or off on the 80B and the 35B.
• --experts N asks the router for N experts per word instead of the model's own number. Fewer experts means fewer bytes per word and faster writing; the answers change, so the quality check is beside the speed on the models page.

Fits in memory

• Wrangler. Coachwhip now watches its own memory so a 16 GB Mac is not pushed over. At start it reads the GPU limit macOS gives it and the model's own sizes, and lowers the bank if a long chat would not fit, saying so. The 122B is tested to a 12,000-token prompt. After every answer it warns if macOS memory pressure is high.
• The bank is allocated at its exact size. Buffers were rounded up to a power of two, which wasted over 1 GB on the 122B; the room now goes to more expert slots.
• The word table stays packed. It used to be unpacked in full at load (a 12 GB peak on the 122B); now each word unpacks only its own row. About 1 GB less memory, no change to the answers.

Fixed

• Fast mode could compute a word with the wrong expert, rarely and silently: a stale pick could pass the CPU's check. Each pick now carries its word's stamp.
• The expert banks are no longer built while a prompt is being read. On a long first prompt that could push the GPU out of memory.

Structure

• New crate crates/prepare: the squeeze, streaming, so the whole model never sits in memory.
• crates/engine/src/requant.rs: Metal kernels that squeeze a Q3_K, Q4_K or Q5_K expert to Q2_K on the GPU, one thread per 256-weight block; the prepare step uses them.
• Vendored candle-metal-kernels with the Q2_K matvec fix (the upstream kernel read 4 rows per group where the shader covered 8) and the shared-event wait the parallel path needs.
• Lab switches behind COACHWHIP_* environment variables are off and inert; they are not part of the product.

Not tested

• 0.4.0 is measured on the 122B and the 80B. The 35B is unchanged in code but was not re-measured on this build.
• Macs other than an M4 with 16 GB; M1 to M3 will be slower.
• The 122B beyond a 12,000-token prompt.
• Exact mode at 6,000 tokens; exact mode is measured at 12K (2.2 tok/s).
• The 122B's sampled answers beyond 300 words.
• prepare on a Mac without a Metal GPU falls back to the CPU and takes about 12 minutes; not timed on this release.
• If the disk fills during prepare, the half-written file stays behind (it will not load; delete it by hand). Fix planned for 0.4.1.

Risk

• Fast mode is a different model in practice: 4 experts where the model expects 8. On six agent coding tasks through Aider (Python, TypeScript, Rust, hidden tests), fast mode passed 5; the sixth, which had a built-in contradiction, it argued with itself until it ran out of room, and exact mode solved it in one go. The README and the chat label it; the quality numbers on the models page are the measure of what it costs.

Coachwhip 0.3.0

Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Method and every figure's conditions: docs/MODELS.md.

Faster

• The early guess. While the GPU works on one layer, Coachwhip runs the router of a layer four steps ahead on the same input and reads the experts it names while the GPU is busy. A guess that would land too late is dropped instead of evicting an expert in use. Defaults --prefetch 10 --ahead 4; --prefetch 0 turns it off. Answers are byte-identical with it on or off.
• Q3_K_M recommended for the 80B, bank 70: smaller experts read sooner, and the same bank memory holds more of them.
• Qwen3-Coder-Next, five coding prompts, 300-token cap: Q3_K_M at bank 70, 5.4 to 5.7 tok/s over three runs (0.2.0: 4.5); Q4_K_M at bank 56, 4.1 tok/s (0.2.0: 3.4). Quality on 8 hidden-test tasks: 7/8 for both files, the same as 0.2.0.

New model

• Qwen3.6-35B-A3B (qwen35moe): Qwen's newest MoE generation. 40 layers, 256 experts, 8 used. Runs from the unsloth UD-Q4_K_M file (22 GB) at --bank 44: 6.2 tok/s over the five prompts, 8/8 on the quality tasks, answers byte-identical across speed settings. Its linear attention pairs value heads with key heads differently from Qwen3-Next, and the maths was checked layer by layer against llama.cpp on the same file and prompt.
• A thinking switch, for models that have one: the "let it think first" box on the chat page, --think in the terminal, always off through the API. It only puts the model's own template text at the start of its turn.
• Bank 56 also runs the 35B at the same speed but holds 2 GB more; on 16 GB it ran out of GPU memory once on a long thinking answer. Use 44.

Structure

• Cargo workspace: crates/engine, crates/server, crates/cli.
• 59 unit tests (cargo test --release) and two Mac checks in tests-mac/: answers identical across speed settings; every supported model loads. Qwen3-Coder-30B-A3B runs with the early guess too (7.6 tok/s at bank 56, one run by the tester).
• bench/speed and bench/quality: the scripts behind every number on the models page, so you can run them on your Mac.
• README: a conversation holds up to 16K tokens.

Not tested

• Qwen3.6-14B-A3B (community cut of the 35B): same architecture, not run.
• Qwen3.5-122B-A10B: needs a split-GGUF loader and more disk; later.
• Macs other than an M4 with 16 GB.

Risk

• The thinking switch changes the prompt bytes only for models whose template has a thinking mode; the 80B's prompt is unchanged (checked byte-identical against 0.2.0's answers).

Signed

Tested and signed at c18ddc4 (binary sha256 c2a17d75e533ae0e79702d9d9716ba5b3a94a72ae7feb2269722a06607605331, built by install.sh from this tag). No stranger has installed this yet: if you do, tell us how it went in an issue.