colibri v2.1.0

Huge open models on the machine you already have: on the CPU, on any GPU through Vulkan, on NVIDIA cards through CUDA. Pure C, one file per model family.

387 pull requests since v2.0.0, 208 of them from 35 contributors. GGUF files now run natively on every engine, read exactly as published. Four new models arrive, Mistral-Large-4 (1T) among them. CUDA gets device chains of its own and a routed-expert tier for every engine. Qwen3.8-Flash-Next runs a 256k context at the speed of 64k, an agent's turns read only what is new, and there is a new chat UI.

Highlights

• GGUF on every engine. coli import takes a GGUF where it lies and every engine runs it as published: no conversion, no second quantization, from IQ1_S to Q8_0, F16, BF16 and MXFP4, the same blocks and the same tokens on the CPU, Vulkan and CUDA. unsloth's Dynamic UD quants run as they are (docs/gguf.md).
• New models. Mistral-Large-4 (1T, 52B active: text, pictures, reasoning, tools), Muse Spark Open (816B), Flint (40B, 2.5B active), LFM3-VL-2B with its vision tower, and the published DeepSeek V4.1 Flash and Kimi K3 GGUFs.
• CUDA. Every layer of Qwen3.6, Qwen3.8 and OLMoE as one chain on the card, a routed-expert tier on the card for every engine, the dense matrices on the card for DeepSeek V4, V4.1, Kimi K3, GLM-5.3 and MiMo, and a GGUF's blocks on CUDA with the prompt on the tensor cores.
• Vulkan. A model larger than the card keeps the experts the card cannot hold in a host pool in RAM, and the CPU computes them inside the device's frame while the card computes its own. Prompts on the matrix units, flash attention with windows and sinks, float16 KV mirrors.
• Long contexts and agents. Qwen3.8-Flash-Next at 262,144 tokens: decode 9.6 to 45.7 tok/s, prompt 15.6 to 149. A short request between two turns (a title, a summary) parks the conversation instead of taking its state, on every engine: a Hermes Agent turn went from 86 s to 4.9 s.
• A new chat UI. coli ui: reasoning in its own block, pictures, web search, a live map of the experts as they fire, a monitor, your models, and coding agents already pointed at colibri. Seven languages.
• Speculative decoding. Qwen3.6's MTP head on every backend (Q36_MTP=1): decode 99.7-104.5 to 115.8-120.1 tok/s on CUDA, 83.2-83.8 to 99.9-109.3 on Vulkan; DeepSeek V4.1's DSpark head drafts as deep as its confidence says, behind a gate that skips the drafts that cannot pay.
• Memory. KV caches in float16: GLM-5.3 at 131,072 tokens in 11.9 GB instead of 23.9.

Measured on one 24 GB card

An i5-13500 with 62 GB of RAM and an RTX PRO 4000 Blackwell SFF (24 GB, 70 W), each model served by coli serve: a fresh 450-token prompt, then a 128-token streamed answer, the request's HTTP included; the better of two runs, tok/s.

GGUF	CUDA prompt	CUDA decode	Vulkan prompt	Vulkan decode	CPU prompt	CPU decode
Qwen3.6-35B-A3B UD-Q4_K_M	3,491	104.8	2,076	92.9	122.6	13.9
LFM3-VL-2B Q4_K_M	12,317	235.1	8,174	214.1	286.5	34.5
OLMoE-1B-7B-0125-Instruct Q4_K_M	10,418	354.6	6,867	299.4	279.9	47.4
LFM3-VL-2B F16	11,524	103.4	7,767	99.3	49.7	6.4
Qwen3.8-Flash-Next IQ2_XS (68 GB: the experts the card cannot hold stay in RAM)	217.1	51.4	214.3	50.6	38.3	9.3

Bigger than the RAM, on the same machine

Each model from its published GGUF; what neither the card nor the RAM holds comes from two NVMe drives. Tokens per second, from each model's page in docs/.

GGUF	size	backend	prompt	decode
Inkling-Small UD-IQ2_XXS	77 GB	Vulkan	15.1-33.1	10.92-11.53
MiMo-V2.6-Flash Q2_K_S	106 GB	Vulkan	8.2	7.8
DeepSeek V4 Flash 0731 UD-IQ2_XXS	85 GB	Vulkan	19.7-20.4	6.18-6.25
DeepSeek V4.1 Flash UD-IQ1_S	233 GB	Vulkan	31.8 over 3,403 tokens	5.01-5.29
GLM-5.3-Flash UD-IQ2_XXS	95 GB	Vulkan	9.3	2.92-3.40
Muse Spark Open UD-IQ1_S	176 GB	Vulkan	5.0-5.3	1.55-1.58
Mistral-Large-4 UD-IQ1_S	233 GB	CUDA	240 tokens in 66-82 s	1.34-1.36
GLM-5.2 UD-IQ1_S	202 GB	Vulkan		0.633-0.658
Kimi K3 UD-IQ1_S	594 GB	Vulkan	31 tokens in 46.5 s	0.349

Run a GGUF

coli import model.gguf                    # the first part of a split file, its other parts next to it
coli chat --model model.colibri           # or coli serve, then coli ui

unsloth's DeepSeek V4.1 Flash, for example:

hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1_S/*" --local-dir v41
cd v41/UD-IQ1_S
coli import DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf
COLI_VULKAN=1 coli chat --model DeepSeek-V4.1-Flash-1017-UD-IQ1_S.colibri

Downloads

file	for
colibri-v2.1.0-linux-x86_64.tar.gz	Linux x86-64 (AVX2), every engine with Vulkan built in
colibri-v2.1.0-windows-x86_64.zip	Windows x86-64 (AVX2), every engine with Vulkan and the AMD NPU lane; coli.cmd to start
colibri-v2.1.0-windows-x86_64-cuda.zip	NVIDIA cards on Windows (compute 8.0 and newer): unpack it over the Windows archive
colibri-v2.1.0-macos-arm64.tar.gz	Apple Silicon; colibri, inkling and kimi_k3 with Metal
SHA256SUMS.txt	the checksums of the archives

The one-step setup (START-HERE.bat on Windows, ./start-here.sh on Linux and macOS) picks the archive for your machine, or builds from source when a compiler is there. Every archive was unpacked and checked by CI before publishing: every engine present, coli dispatching to each one, the Vulkan engines finding their shaders, the dashboard and the chat UI loading.

What changes by default

• The KV caches are float16: colibri.c's MLA latent cache (GLM-5.2/5.3, Mistral-Large-4; KV16, not with CUDA or Metal), qwen36's (Q36_KV=f32 keeps float32) and qwen38's when serving (Q38_KV). Half the RAM of float32.
• A short request between two turns parks the conversation on every engine serving one conversation (COLI_PARK=0 never parks).

Thank you

To the 35 people whose pull requests are in this release: @rudycelekli, @CoTi1938, @bobsummerwill, @Avicennasis, @appleguy, @Stamina9, @kreuzzelg, @simaotwx, @Altroo, @benmaster82, @namespaceMarcello, @JackKnifeAI, @leonwieser2009-oss, @Ontahm, @Varshith-Kali, @djairjr, @neerajdad123-byte, @dajiaohuang, @mfranzon, @huppiflupp, @seanfalling, @jameswitney, @KhalidAlnujaidi, @HyperionBurn, @TyrionV8, @kflirik, @bherald, @Anionex, @mfethe1, @ZacharyZcR, @ggeorgovassilis, @neoo69, @crichalchemist, @enitimeago and @pavolbauer. And to everyone who opened an issue with numbers from their own machine.

Everything in 2.1.0

New models

• #2493, #2500, #2503: Mistral-Large-4, the text model from every
  published GGUF and from the FP8 release, on the CPU, the Vulkan chain and CUDA,
  and its pictures (the Pixtral tower, from the GGUF's mmproj or our containers).
• #2489, #2491: LFM3-VL's text model (from its GGUFs or the converted
  checkpoint) and its pictures.
• #2292, #2297, #2310, #2316: Flint, its safetensors and every
  unsloth GGUF, on the CPU, Vulkan and CUDA; its sliding layers keep their K/V in
  a ring.
• #2293, #2324: Muse Spark Open from the unsloth GGUFs, CPU, Vulkan and
  CUDA.
• #2143, #2269: DeepSeek-V4.1-Flash-1017 as it is, and the published
  DeepSeek V4.1 GGUFs with their DSpark head.
• #2274: the published Kimi K3 GGUFs, on the CPU, the Vulkan chain and the
  tier.
• #2102: Qwen-Image gets a single-frame CPU VAE encoder.

GGUF on every engine

• #2130: GGUF models read natively, the CPU kernels interleaved by 16 rows.
• #2140, #2141: DeepSeek V4, Inkling, GLM-5.3 Flash, GLM-5.2, MiMo,
  DeepSeek V4.1 and OLMoE read GGUFs too.
• #2182, #2185, #2195, #2228, #2237, #2238: a GGUF's
  dense matrices and routed experts on the Vulkan chain, its prompt on the
  device's tiled and grouped GEMMs and on the matrix units; IQ1, IQ2, IQ3 and
  K-quant block decoders.
• #2252, #2254, #2255, #2491: a GGUF's block formats on CUDA,
  Qwen3.6's and Qwen3.8's CUDA chains on a GGUF, the prompt GEMM on f16
  mma.sync.
• #2239, #2248: on an NVIDIA card a GGUF runs on Vulkan where CUDA does
  not cover its work (GLM-5.2, DeepSeek V4, Kimi K3, Inkling).
• #2503, #2515: colibri.c's GGUF experts and dense matrices on CUDA, and
  F16 and BF16 matrices on the GPU as block formats (Qwen3-0.6B F16 on CUDA:
  prompt 137 to 15,921 tok/s).
• #2198, #2280: IQ2_S and IQ3_S rows in one gather; expert reads without
  readahead (Kimi K3 decode +30 %).

CUDA

• #1992, #2089: every layer of Qwen3.6 and of Qwen3.8 as one chain on the
  CUDA device (COLI_CUDA_CHAIN).
• #2209, #2231, #2232, #2233: the routed experts in the chain's
  frame, a prompt chunk's attention and GEMMs on the tensor cores, the experts
  outside VRAM staged from the RAM cache.
• #2276, #2277, #2278, #2279: a routed-expert tier on a CUDA
  device for every engine (cu_tier), and the dense matrices on the card
  (cu_dense.h) for DeepSeek V4, V4.1, Kimi K3, GLM-5.3 and MiMo.
• #2257, #2332, #2377, #2395, #2416, #2417, #2423:
  the card's residents follow the conversation, promoted from the host pool.
• #2335: the Vulkan and CUDA chains serve several conversations at once, each
  one's state on the device.
• #2459, #2461: the decision engines (Laya, GLiNER2.5-Decide) and OLMoE,
  every layer with its experts, on a CUDA card.
• #2462, #2497, #2385, #2392, #2408, #2418, #2421,
  #2453: progra...

colibri v2.0.0

169 pull requests since v1.12.1, 88 of them from contributors. Every MoE
engine now runs on any GPU a Vulkan driver can see: the routed experts on a
shared device tier, the dense layers on a device chain (all of them, or the
first N that fit), and a second GPU for more experts. Three model families
arrive (MiMo-V2.6, Qwen3-Coder, Qwen-Image-2.1) and one dense checkpoint
(Qwen3.8-27B). Qwen3.8 Flash Next gets int4 experts and speculative decoding
that is on by default. Every text engine serves up to 16 conversations at once.
The release archives carry the GPU backends. Setup is one step, and there is one
decision API (System One).

Several conversations at once

• #1955: every text engine serves several conversations at once (coli serve --kv-slots N, up to 16), as GLM-5.2 and GLM-5.3 Flash did: qwen36 (Qwen3.6,
  Qwen3-Coder, Qwen3.8-27B), qwen38, OLMoE, Inkling, Kimi K3, MiMo-V2.6,
  DeepSeek V4 and V4.1. Each slot keeps its own conversation's state, and every
  decode step takes the next token of each active request as one batch, so the
  weights and experts a step reads serve all of them. A request gets the tokens
  it gets alone, checked frame for frame on every engine, on the CPU and with
  the expert tier. With more than one slot nothing drafts and the dense chain
  stays off.

Vulkan on every engine

• #1830: every engine's resident matrices run on any Vulkan GPU (fmt 10, 11
  and 12 added), each configuration checked against the engine's own CPU run on
  Lavapipe (tests/vulkan_engines.sh).
• #1834: tiled GEMMs for prefill in every weight format, and a
  cooperative-matrix GEMM where the device has one.
• #1840, #1843: a shared routed-expert tier (vk_tier.c). It holds a
  cache of experts in device memory, filled from the expert history and adapting
  while you chat, and computes the resident experts of each step as one async
  batch while the CPU computes the rest. First on qwen38 and qwen36, then on
  every MoE engine: GLM-5.2, GLM-5.3 Flash, Inkling, OLMoE, Kimi K3, MiMo-V2.6,
  DeepSeek V4 and V4.1.
• #1844, #1848, #1849, #1851, #1857: the dense chain
  (vk_chain.c), a layer's dense part in one submission, on qwen36, qwen38,
  MiMo-V2.6 (sliding window and sinks on the device), OLMoE, Inkling (and
  OLMoE's PILOT race fixed), GLM-5.2 and GLM-5.3 Flash (shared MLA, DSA, KDA and
  mHC ops), Kimi K3, DeepSeek V4.1 and DeepSeek V4.
• #1871: staged device-local uploads for cards without Resizable BAR, where
  the mapped path failed or spilled to system RAM.
• #1901: expert streaming for big prompt steps, memory budgets and
  speculative verification on the device.
• #1909, #1913, #1916, #1918: a partial dense chain. The first N
  layers that fit go to the device, the CPU runs the rest, and nothing is
  uploaded that the chain will not use, so an 8 GB card with a larger trunk keeps
  the chain and its expert tier instead of losing both. COLI_VK_CHAIN_LAYERS
  forces N, coli plan predicts it. DeepSeek V4, qwen36 and olmoe, qwen38,
  GLM-5.2 and GLM-5.3 Flash.
• #1920: a second GPU for the expert tier (COLI_VK_DEV2=auto|<index>) on
  every MoE engine: the experts after the primary device's, each step one batch
  per device, both in flight at once. A failing second device gives its experts
  back to the CPU and the tier goes on with the first.
• #1942: the dense chain's layers on two GPUs. With COLI_VK_DEV2, the
  layers a partial chain leaves go to the second device instead of the CPU, on
  every chain engine (qwen36, qwen38, OLMoE, MiMo-V2.6, Inkling, GLM-5.2,
  GLM-5.3 Flash, Kimi K3, DeepSeek V4.1 and V4). COLI_VK_CHAIN_LAYERS2 forces
  how many, COLI_VK_CHAIN_DEV2=0 keeps them off. A lost second device leaves
  the CPU to run its layers, the recurrent state rebuilt where it was held.
  Checked on Lavapipe opened twice; two real GPUs not measured yet.
• #1933: COLI_VK_KV_COLD=device (opt-in): with the KV cache split past the
  device's budget, the host's part of the attention runs on the device too, from
  host memory it reads in place, so a step runs in one frame. The prompt gains
  4-6% on a Radeon 780M, decode is slower there (hence off by default); a
  dedicated GPU has not been measured.
• #1948: several conversations at once on the dense chain. colibri's
  multiplexed decode (KV_SLOTS, one row from each active conversation) runs on
  the device: the batch's matrices as one, each row's attention over its own
  conversation's KV mirror (up to COLI_VK_CHAIN_MUX, default 16, beside the
  chain's own) with its own DSA list; two devices and the partial chain too.
  Every frame checked against the CPU's on Lavapipe; not measured with a real
  checkpoint yet.
• #1934, #1935, #1936, #1937, #1938, #1939 (@huppiflupp):
  the matrix units for prompt attention and int8/int4 prompt GEMMs on the
  chain, a step's experts as one grouped GEMM (decode: one grouped GEMV per
  phase), qwen36's routing in parallel, and int8 decode GEMVs sized by the
  matrix. On a Radeon 780M with Qwen3.6-35B-A3B the first token of a
  1000-token prompt comes in 9.3 s instead of 12.6, decode 6.1 tok/s instead
  of 5.7, perplexity unchanged.
• #1954: on an Intel GPU the dense chain decoded garbage (Qwen3.6 on an Iris
  Xe, the default there). Its pipelines now run at the subgroup size the device
  reports; the device compiles at 8 to 32 lanes, and the chain's GEMVs had run
  at another width than the one they read. On Windows the chain also finds its
  optional shaders (MLA, KDA, mHC, DeepSeek V4, the split KV cache), and the
  split cache frees its host memory with the matching call.
• #1911: COLI_VK_DEV=<index> picks the device on a machine with two GPUs of
  the same kind; the cooperative GEMM's epilogue stride is aligned to 16 bytes, a
  validation error on some drivers (#1908).
• #1846 (@crichalchemist): correct expert-batch results under MoltenVK's
  default mode.
• #1762, #1763 (@Kenneth-Javier): the Vulkan loader linked on Windows
  under -static, and the Vulkan objects linked into every engine-including
  test rule.

Models

• #1813, #1819, #1820, #1827, #1835: MiMo-V2.6 (Flash and
  Pro): the engine, the vision tower, tool calling and Brio mode with logprobs,
  read as released; Pro verified against Xiaomi's reference on the real
  checkpoint, and coli serve names the loaded variant.
• #1828, #1829: Qwen3-Coder-30B-A3B (qwen3_moe) on the qwen36 engine.
• #1777, #1779: dense checkpoints of the Qwen3.6 family on the qwen36
  engine (Qwen3.8-27B), text and images.
• #1780: Qwen-Image-2.1, text to image: a C engine, pictures in the
  terminal, the web app and the OpenAI images API.
• #1951: Qwen-Image on the GPU and a faster start. The weights load in
  7.5 s instead of 34 (four layers at once, quantized straight from bf16, the
  same int8 bytes). With Vulkan every block of a denoising step runs on the
  device, its attention included, and the VAE decodes there too; a card that
  does not hold the 7.1 GB transformer keeps the blocks that fit and streams
  the others each step. On a Radeon 780M a 512x512 image takes 61 s instead of
  112 and a 1024x1024 step 38 s instead of 67; every stage checked against the
  diffusers reference.
• #1831: Qwen3.8 Flash Next's routed experts as int4-g64: 1.4 to 1.56x
  decode.
• #1832, #1917: speculative decoding with the checkpoint's MTP head,
  lossless, +12-14% on the release; with #1917 it is on by default with prompt
  lookup on every engine that has it (Q38_MTP=0, COLI_LOOKUP=0 turn them
  off), and coli plan counts the MTP head's memory.
• #1814: Inkling reads the embedding norm under the name transformers 5.18
  writes.

System One: one decision API

• #1856: POST /v1/systemone, one decision API, and the Laya decision
  engine; /v1/brio is removed.
• #1868: GLiNER2.5-Decide as a native decision engine (DeBERTa-v3 and
  GLiNER2's head).
• #1872: Clef as a native decision model on qwen36, a joint head over
  Qwen3.8-27B, with a 16-bit dense mode.
• #1743: the brio serve gates run in CI against the converted fixture.
• #1753 (@tarazum): brio normalizes to sum by default and warns when a mean
  mixes option token counts.

Setup and the CLI

• #1953: the release archives carry the GPU backends: every Linux and
  Windows engine built with Vulkan and its shaders, the macOS engines with
  Metal, and a Windows CUDA package (coli_cuda.dll for compute 8.0 and newer,
  with colibri, qwen36 and kimi_k3 built to load it). The Vulkan loader is
  opened at run time, so these engines still start on a machine without one,
  on the CPU. coli setup uses them without a Vulkan toolchain.

• #1842: one-step install and start (start-here, coli setup), the
  AI_SETUP.md guide and an MCP server.

• #1850, #1896: setup picks the backend that works on the machine: on an
  integrated GPU, Vulkan only where it was measured faster; CUDA only when the
  toolkit builds for the card, Vulkan otherwise, and a failed build falls back.

• #1903: an engine already built is brought up to date with make before it
  is used (a git pull used to keep running the old binary), and coli logs
  says where a foreground server prints (#1852).

• #1919: a fresh install of Qwen3.6-35B-A3B or Qwen3.8 Flash Next gets a
  starting expert history, so its first run fills the GPU tier from the start:
  on a Radeon 780M Qwen3.6's first run went from about 6 to 11 tok/s.

• #1817: in coli chat, a line that starts with a picture's path is a
  message, not a command.

• #1959 (@rudycelekli): coli setup counts a server on its port as a running
  colibri only when /health answers as colibri does; another program's HTTP 200
  no longer passes for one, and the setup starts its own on the next free port.

• #1315 (@Avicennasis): resuma...

colibri 1.12.1

96 pull requests since v1.12.0, 80 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, coli chat working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.

Tokenizers, measured against the reference

• #1654: qwen36 tokenized differently from HF tokenizers in two ways.
  An added token right after punctuation was encoded as text (X.<|im_end|>
  was 7 tokens instead of 3, every chat turn ending in punctuation paid +4,
  #1653), and a whitespace run followed by a non-space was one piece where
  the regex's \s+(?!\S) leaves the last char to the next one, so every
  indented line of code tokenized differently. Measured on the real
  vocabulary: 2,803 lines and blocks of code, Markdown, Chinese and
  Japanese went from 757 identical to 2,803, with 5.4% fewer tokens.
• #1656: OLMoE's tokenizer.json has no Split, a bare ByteLevel with
  use_regex, for which HF runs the original GPT-2 pattern; tok.h applied
  cl100k. A GPT-2 family in tok.h: 1,560/1,708 identical before, 1,708/1,708
  after. The same measurement on GLM-5.2/5.3/5.3-Flash, DeepSeek V4 and V4.1,
  Inkling and Qwen3.8 came back identical on every case.

Brio and the serve contract

• #1662: POST /v1/systemone, the request and the reply of TypeSafe's
  Jev API, served by the brio channel: a client written for it points at
  colibri and changes the base URL. noul is a yes/no question, choice
  scores the labels with their descriptions in the text, score the level
  numbers with the expected value and the legend; confidence by their
  documented formula. Any model name is accepted on that route. Measured on
  the real Qwen3.6: the three-question example of the docs in 1m46 with the
  state read once.
• #1655: the DeepSeek V4 engine speaks the numeric channel (logprobs=k,
  pin=1, max_tokens=0), so /v1/brio works on the ninth engine instead
  of answering 500 (#1648). The head that used to keep only its argmax now
  returns the whole row; ECHO per prompt position during prefill, the
  prompt-end scores kept with a state snapshot on pin, and a logprob tail
  on every DATA frame during generation. The tiny fixture pins that the
  best ECHO token equals the greedy token from the same prefix, and that
  the pinned predictor equals a cold prefill's.
• #1659: qwen36 and qwen38 refused a request when prompt + max_tokens
  exceeded the context, and the gateway's default budget for these two
  families is 8192, the whole default context: every request without
  max_tokens and every coli chat message answered 400 on a two-token
  prompt (#1641). max_tokens is now a ceiling, clamped to the room the
  prompt leaves, as GLM and DeepSeek V4 already did; only a prompt that does
  not fit is refused. docs/api.md states the rule, docs/qwen38.md names
  Q38_MAXT as the variable --ctx becomes.

The dense trunk in VRAM, measured before it is placed

• #1657: qwen36 offers the rest of its dense trunk to the VRAM placer:
  the DeltaNet out_proj (dnout), the attention q/k/v/o (attnproj) and
  the shared expert (shexp), about 650 MB more of int8 on the 35B beside
  lmhead and dnproj. Measured on four Tesla M10 by the reporter of
  #1652, every placed component ran slower than the CPU (lm_head 68.8 ms
  against 41.7), so the engine now times one GEMV both ways at startup and
  withdraws the whole automatic placement when the GPU loses, giving the
  VRAM back to the experts: auto equals off on that box, byte-identical
  output. A hand-written COLI_PLACE stands; COLI_TRUNK_PROBE=0 trusts
  the placer.

Performance

• #1664: qwen36's dense trunk and routed experts multiply with integer
  dot products. The activation is quantized to int8 once per call and the
  weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
  instead of a float conversion per weight; the integer kernels move from
  quant.h into idot.h, shared by every engine. Measured on the 35B, 8
  threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head
  12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3%
  perplexity on 4 x 512 tokens. Both are the default (COLI_DENSE_IDOT=0,
  QWEN_EXPERT_ACT=f32 restore the f32 kernels). COLI_DENSE_BITS=4 with
  COLI_DENSE_INT4=<components> stores part of the trunk as int4 in blocks
  of 64: opt-in, with the perplexity it costs per component in the docs
  (lm_head alone +2.4%, everything +10%).
• #1668: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of
  BF16 read on every token, more than the ten routed experts) is kept on the
  CPU as int8 rows with the BF16 copy released, and multiplied with the same
  integer kernels; the routed experts' e4m3 blocks are decoded eight at a
  time in registers and multiplied with FMA instead of one table lookup per
  weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM
  LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,
  lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;
  prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%
  (two chunks lower, two higher); the vector FP8 kernel alone reproduces the
  BF16 run to four decimals. Both are the default (Q38_TRUNK_CPU_INT8=0
  keeps the BF16 trunk, Q38_FP8_KERNEL=scalar the table kernel).

Performance, from contributors

• #1606: the K1b grouped int4 family gets a multi-row tile and AVX-512
  and AMX arms, and is no longer switched off on AVX-512 builds; exact on
  all eight engines on a 16-core AVX-512 host.
• #1239: an SSE4.1 tier for the olmoe and qwen36 int8 GEMV, for hosts
  with SSE4.1 but no AVX2; on the Sandy Bridge of #1652 decode went from
  2.45 to 3.54 tok/s.
• #1313: matmul_fp8 computes four output rows per pass under clang,
  where the contraction makes it bit-exact; GCC keeps the one-row kernel.
• #1612: qwen36 gains the GLM engine's CACHE_ROUTE lever, with the
  VRAM tier as the first residency level, opt-in.
• #906: DEGRADE_ZERO, an opt-in policy that zero-fills a missed
  expert slot below a gate-weight threshold instead of blocking on the
  load (#865).
• #1677: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the
  card in blocks of up to 256 rows instead of one row at a time; a 259-row
  prefill makes 2 projection calls instead of 259, with the convolution
  history and the recurrent state checked against the CPU run. The paired
  microbenchmark on an RTX 4070 read 4 to 10x per projection.
• #1674: qwen36's attention projections the tier placed in VRAM answer
  a whole prompt batch with one call per matrix; a failed call turns only
  that handle off and the prompt continues on the CPU.
• #1676: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU
  intermediates on the device and applies down there (one fused entry
  point, optional in the DLL: an older backend keeps the three-call path).
• #1673: the streaming MXFP4 matmul reuses one grow-only device scratch
  per card instead of allocating and freeing weights and scales on every
  call.
• #1559 (kreuzzelg): convert_qwen36.py --down-bits 8 writes the mixed
  expert layout, int4 gs64 gate/up and int8 down in one slab (5.7 bits per
  weight against gs64's 4.5); the engine tells it apart by size and reads
  each matrix in its own format on the CPU path, and refuses the VRAM tier
  with a line. It is the knob behind the #1370 numbers: on wikitext-2 the
  int8 down alone recovers a quarter of the gap between gs64 and all-int8,
  the rest sits in gate/up. A measurement tool and a middle step, not the
  answer to the gap.
• #1730 (mfethe1): DeepSeek V4's FP4 expert kernels, the prefill batch
  and the decode matvec, get a NEON arm; arm64 used to take the scalar
  arm, which is why Apple Silicon prefilled at decode speed (#1696). Bit
  identical to the scalar arm, and the ARM CI job now checks that on every
  change; 17 to 22x on the kernel at the V4 expert shapes on an M-series
  Mac, as measured by the author.
• #1716 (jtinbergen): qwen36 quantizes its dense weights to int8 while
  loading instead of keeping an f32 copy first, and converts f16/bf16 with
  SIMD. On the 35B the resident set after load goes from 9.2 to 4.8 GB;
  the generated text and the perplexity are identical to before.
• #1286 (cameron): the grouped int4 GEMV and the fused gate/up GEMV get
  an SSE4.1 arm for CPUs without AVX2 (Ivy Bridge and older). It vectorizes
  across output rows, so each lane runs the scalar row's exact sequence and
  the result is bit-identical; forced-SSE4.1 tests at -O1, -O3 and without
  FP contraction pin it. 2.2 to 2.7x on the isolated kernel on a dual
  E5-2680 v2.
• #1686 (DebugSultan): qwen38's prefill chunk (Q38_PREFILL_BATCH_ROWS)
  and workspace (Q38_PREFILL_WORKSPACE_MIB) are runtime knobs, the expert
  load batch is no longer capped at top-k, and the QSA ranking and
  attention run per position in parallel at prefill. Measured on the
  released checkpoint on top of the int8 trunk: output byte-identical at
  every chunk width, no speed change on our 16-core server; the knobs are
  there for hardware where the chunk binds.

Fixed

• #1650 (bokiko): a Qwen3.8 pin snapshots the recurrent and PLE state
  but reuses the live attention and indexer rows; after an unrelated prompt
  overwrote those rows, returning to the pin could change brio logprobs
  without a warning. The engine now records the token identity of the live
  rows (kv_prefix.h) and refuses a stale pin or prefix restore; image rows
  are tainted. Wire regressions run on the BF16 and FP8 fixtures.
• #1626: SNAP is the model directory for every non-GLM engine, so
  coli run stops handing them a leftover environment (#1600).
• #1604: glm53 honours `Mat...

colibri 1.12.0

colibri 1.12.0

81 pull requests since v1.11.0. A new way to ask a model a closed question,
a redesigned dashboard and landing page, and a long run of small failures
that used to answer 500 or die on a locale.

Brio mode: score a closed set instead of generating

• #1632: brio mode, on all nine engines. Hand the engine the options an
  answer is allowed to take and it reports the probability of each one
  instead of writing the answer: the option tokens are read, not sampled,
  so completion_tokens is 0 and no reply can fall outside the list. It is
  opt-in per request through two SUBMIT keys, logprobs=k and pin=1;
  without them every frame is byte-identical to before, which is asserted
  per engine rather than claimed. max_tokens=0 became legal, but only
  together with logprobs>0.
• The shared contract is three headers: decode_batch.h for the logprob
  tail, serve_codec.h for the wire, and pin_pool.h for nested state
  snapshots, so a request can keep one photograph of the shared
  instructions and a deeper one of the instructions plus the question. With
  one level the question is re-read once per option; with two, four items
  cost 176 tokens instead of 496.
• kv_prefix_holds() is the guard that makes the snapshots safe: a
  snapshot is reused only when the prefix record still holds the same ids,
  because a K/V bank can be dropped between two options.
• Exactness is measured, not asserted: the logprobs read from a snapshot
  match a cold recomputation to 0.00e+00 on all nine tiny fixtures, and on
  the two real checkpoints available (qwen36 at 22 GB and DeepSeek V4.1
  Flash at 476 GB).
• Three clients: POST /v1/brio on the gateway, /brio in coli chat,
  and a page in the dashboard. docs/brio.md has the request, the reply,
  the normalisation choice and the cases where the mode does not help.
• The endpoint takes three forms. options is one closed question.
  questions is many questions on one state, each with its own options,
  with the state photographed once and the snapshots ordered by the server.
  schema is an object of field to allowed values: the server writes the
  JSON skeleton and fills it one field at a time, so the object is valid by
  construction and every value is one the caller allowed, each with its own
  probability and entropy. Both were measured before they were exposed: the
  5.7x and the 2.4x above are these two forms.

The dashboard and the landing page

• #1633: the web dashboard is redesigned around a workspace and a
  navigation dock, with a brio page that reads a document once and asks it
  several questions, each with its own option set, and a light and dark
  theme.
• #1635: the landing page lists the models as a searchable list instead
  of a card grid, gains a brio section, and gets a light and dark theme
  built on the mechanism proposed in #1551. The shipping version and the
  family names are generated from c/version.py and c/family_registry.py,
  so the page cannot claim eight families again while nine are running.
  The hero plays a recorded session rather than a mock-up.

Performance

• #1477: one routed-expert kernel, expert_ffn.h, shared by the MoE
  engines and used by qwen36 first: int4 kept planar in RAM instead of
  unpacked to int8, and two OpenMP regions per layer instead of three per
  expert. 12.8 to 15.7 tok/s with the resident set down from 29 to 17 GB,
  output identical.
• #1424: a CUDA VRAM tier for Qwen3.8-Flash-Next, 43 % on one 8 GB card
  and 60 % on two.
• #1553 and #1558: the previous turn's KV is reused instead of
  re-prefilling the transcript, on qwen36, olmoe and DeepSeek V4.1, with a
  resumed prefill that is exact.
• #1521 and #1524: the mHC mix and the KDA convolution and head
  loops run in parallel, bit-identical, for 3.9 % and 3.0 %.
• #1517: the four engines that never sized their OpenMP team now do.
• #1522: the DeepSeek V4.1 expert stream can be read from more than one
  drive; #1455 weights glm53 expert reads by disk and honours
  COLI_MODEL_MIRROR; #1543 fixes the V4 mirror cache on unified memory.
• #1495: less overhead in the sparse router's top-K selection.

Fixed

• Requests that answered 500 engine failed and now answer properly:
  a non-object json_schema (#1587), an unpaired surrogate escape
  (#1589), a non-object tool_choice.function (#1598), and a tool
  call whose arguments are not an object.
• #1595: the grammar draft walker switched itself off after about sixty
  repetitions. #1596: a nested array schema doubled the compiled
  grammar at every level.
• #1483 and #1514: olmoe, inkling and glm53 refuse an over-long
  prompt with CONTEXT_EXCEEDED instead of failing late.
• #1549: the lazy shard-mapping table in st.h was published without
  synchronisation and glm53 reached it from an OpenMP region.
• #1557: a truncated multibyte tail made the qwen36 tokenizer read past
  the prompt. #1530: an integer overflow check in kimi_k3.c.
• #1631: coli serve reaches the expert-history save when it is stopped
  with SIGTERM, not only on Ctrl-C. #1628: the expert grid is resent
  after every turn, so the dashboard stops showing the cold snapshot
  forever.
• #1608: coli doctor matches core tensor roles by component instead of
  by one family's spelling.
• #1526, #1584, #1585, #1536: the planner, the auto-tuner
  and the registry agree about which engine is running, and stop advising
  GLM-only knobs on every family.
• #1489: coli bench ran the GLM engine on any model. #1503: every
  engine is handed its model directory through SNAP. #1486 and
  #1488: a directory without config.json names no engine, and
  coli convert refuses an output directory that already holds one.

Windows, macOS and locales

• #1515, #1560, #1561, #1562, #1516: the chat, coli run, coli tune and the datapoint harness no longer die when an engine
  line, a prompt or a dash does not fit the console code page.
• #1537, #1579, #1535: the qwen36 CUDA_DLL build on Windows gets
  its tier, is recognised, and the device probe loads the backend before
  counting devices. #1563: the Inkling dense int4 converter finishes.
• #1599: MacPorts libomp built every macOS engine single-threaded.
  #1548: Metal falls back to MTLCopyAllDevices when there is no
  system default device. #1504: olmoe measures available RAM on macOS
  and Windows too.

Diagnostics

• #1494, #1506, #1512: PROF reports block time, head time,
  matmul, attention and lm_head, at microsecond resolution, on deepseek_v4,
  olmoe and qwen36. #1555: CONSIST=1 checks prefill against decode.
• #1482: configurable generation statistics in coli chat, with exact
  token throughput when the server reports it.
• #1468, #1487, #1480: a failure in the MoE step, a missing core
  tensor and duplicate tensor names are diagnosed by name instead of by a
  generic message.

Docs and licence

• #1540 and #1541: the copyright holder is named and a NOTICE file
  is added.
• #1294: a reproducible benchmarking protocol; #1508: a GPU-backend
  datapoint needs a correctness line to be accepted.
• #1471 and #1473: the expired Discord invite is replaced in all
  four READMEs and on the site.

Builds for Linux x86_64, macOS arm64 and Windows x86_64 are attached below by the release workflow, with SHA256SUMS.txt.

colibri v1.11.0

56 pull requests since v1.10.2. A ninth model family, five real bugs closed
across four engines, and the two platforms the C tests never built on now
building them in CI.

A ninth engine: DeepSeek V4.1 Flash

• #1453: DeepSeek V4.1 Flash (552B, 510 GB on disk) runs on a CPU box
  streaming experts from an SSD, with no conversion: the released checkpoint
  is read natively, fp8 dense with 32x32 ue8m0 tiles and fp4 experts whose
  layout is byte-identical to the mxfp4 the Kimi K3 engine already reads.
  Everything in the architecture that is not V4 is in: Engram (two n-gram
  memories of 384M rows, 203 GB, that never enter RAM), the DSA indexer with
  its two-level candidate source, hyper-connections, the compressor, the
  32-layer vision tower with its aligner, DSpark speculative decoding (3
  stages, blocks of 5, verified in one batched forward with a rollback for
  the rejected rows) and tool calling in the checkpoint's own DSML format.
• Held token-exact in CI against a torch-only CPU reference
  (c/tools/dsv41_ref.py, written because the vendor's own forward needs
  tilelang GPU kernels), at three cache capacities, on a short and a
  40-token prompt, under all three speculative modes; the vision tower is
  matched to 5e-06. The CI also asserts that the vendor's index-key
  republication policy and the tidy reading disagree, so the shipped default
  cannot be "corrected" by accident (V41_INDEX_OWNER).
• Measured on the released checkpoint, cold, caches dropped before every run,
  same prompt and seed: a turn went from 78.7 s to 25.1 s during the work
  (0.305 to 0.957 tok/s) through batched expert reads (V41_READ_DEPTH,
  default 8 is the measured knee), attention matrices read once per block of
  positions instead of once per token, and an expert-major MoE. Every step
  is bit-exact against what it replaced. A five-turn chat session runs at
  1.14 to 1.58 tok/s; an image_url part goes through the gateway into the
  vision tower on the same engine.
• Two ways of hiding the expert reads behind the matmuls were built,
  measured worse, and removed; both are written up in docs/deepseek-v41.md
  with the numbers, so the next attempt starts from them.
• coli chat, coli serve and coli web work; coli run is deliberately
  unwired for this family, as for qwen36 and qwen38. Per-turn accounting is
  behind V41_STATS, off by default, so the chat stays clean.

Fixed

• #1390 (@crichalchemist): the qwen36 GPU tier's qt_fill_wait returned
  when the upload queue was empty, but the uploader frees the ring slot at
  dequeue, before the backend copies anything, so the warmstart freed the RAM
  int8 copy of experts still in flight. An in-flight count that the uploader
  decrements only once the slot is resident (#1360).
• #1434 (@njloof): fmt=0 is raw f32 and has no scale array, but the
  Metal sizing helper returned a per-row scale size for it, so
  coli_metal_matmul wrapped a NULL pointer and segfaulted inside
  newBufferWithBytes, or silently produced all-zero output when the size
  happened to be a page multiple. The kernel no longer applies a scale for
  fmt=0, and a fail-closed guard sized off the same helper falls back to
  the CPU path for any format that needs scales and got none.
• #1389 (@bherald): GLM-5.3 teardown leaked the lazily allocated dashboard
  HITS table.
• #1461: coli cluster worker could never start without an explicit
  --cap: the default is None, and str(None) reached the engine's
  argument check as the literal "None" (#1452). The worker now resolves the
  cap the way every other launcher does.
• #1460: olmoe's serve-mode PROF line published five literal zeros, so
  /profile reported expert_disk_s = 0.0 on turns where the expert reads
  were the workload. The disk phase is measured now; the other four fields
  stay zero because they are unmeasured, not because a guess would look
  better (#1449, half of it).
• #1445: olmoe's --ram was inert and "no --cap" meant a constant eight
  slots per layer regardless of memory; the cap is derived from the RAM budget
  (#1443). On the reporter's box a 2,291-token prompt went from 675 s to 353 s
  with no flags (#1442).
• #1377: available RAM is measured on every platform (glm53 read
  /proc/meminfo unconditionally, so on Windows and macOS it read 0 and the
  expert budget collapsed to 1 GB, which is how coli tune went OOM in
  #1375), and the Windows commit limit is respected.
• #1381: the context ceiling is announced, a request that cannot be
  honoured is refused up front, and the number reported is the right one
  (#1376).
• #1393: GLM-5.3 replies always start inside the think block (#1278).
• #1414 (@dajiaohuang): the GLM-5.3 serve loop honours CANCEL while the
  turn is still running (#1332).
• #1423 (@kreuzzelg): the lazy HITS table is built privately and published
  under a lock in qwen36, kimi_k3, glm53 and qwen38; the parallel warmstart
  segfaulted on first touch about one run in twelve (#1422).
• #1404 (@Petsku01): the qwen36 tier offers int8 experts on the decode
  path (#1391). #1388 (@crichalchemist): QT_MAX_ROWS replaces seven
  literal 32s that had to agree, and the int8 copy is freed by ownership
  rather than by format.
• #1421 (@bherald): coli no longer overwrites an explicit
  CUDA_EXPERT_GB when it decides to place the dense trunk on the card.
• #1244 (@Unknown-Findout): the CUDA expert tier is charged real VRAM, not
  logical bytes (#687). #1394: the VRAM prefix is priced per row, not at
  the container's widest (#1351). #1410, #1411: on a single GPU the
  VRAM prefix is sized from the VRAM budget and the measured headroom, not
  from the RAM pin plan capped by the LRU reserve (#1409, #1405).
• #1447 (@trigger2k20): coli mirror verify accepts a complete mirror
  copied by other means, with no receipt, and fails closed on a short or
  corrupt shard.
• #1456 (@trigger2k20): the planner reports memory.unified from the
  host, not from whether the selected engine has a placement-capable GPU, so
  a CPU-only engine on Apple Silicon no longer reads as non-unified; and
  coli tune stops suggesting DRAFT/PIPE/PIN/NUMA to glm53, which
  ignores them.
• #1396 (@dmoraesrs): the web reasoning selector drops "Medium" for
  GLM 5.3, which the engine renders identically to "High".
• #1428, #1435 (@iiEliJas), #1433 and #1432 (@texasich),
  #1440, #1459, #1458 (@trigger2k20), #1463: setenv visible
  to getenv in-process on Windows, malloc_trim guarded to glibc so the
  engines build on musl, a missing bench recipe restored, the per-test
  _putenv_s helpers dropped, tests including compat.h directly, and
  Clang warnings cleaned up.

Added

• #1462 (@trigger2k20): an opt-in Metal path for GLM-5.3-Flash's routed
  experts on Apple Silicon (COLI_METAL=1): gate, up, clamped SwiGLU, down
  and the route-weighted scatter on the GPU, CPU path unchanged when Metal
  is off. Validated against the CPU implementation to 2.8e-09 and measured
  on an M4 Max: 0.469 to 0.621 tok/s at 32 tokens, up to 2.15 tok/s with
  the expert cache tuned. Also fixes coli run --cap N being silently
  ignored on the glm53 one-shot path.
• #1457 (@trigger2k20): GLM-5.3-Flash feeds the shared routing telemetry
  and writes .coli_usage, so usage-driven placement and partial-mirror
  planning have data for it.
• #1454 (@yuripourre): the qwen36 GPU tier compiles with HIP=1.
• #1399 (@SebaWag): TRUNK_RESIDENT_LAYERS=N streams the dense trunk
  through mmap (#826).
• #1361, #1374 (@kreuzzelg): the qwen36 dense trunk places itself
  (COLI_PLACE=auto, priced in bytes saved per byte of VRAM), fp8 streaming
  mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU.
• #1383, #1385, #1386, #1387: Brain and Profile tabs (EMAP,
  HITS, PROF) on every engine. #1382 (@dmoraesrs): a reasoning depth
  selector in the web UI (#1311).
• #1407, #1408, #1406: CI builds the portable Windows CUDA DLL and
  publishes it as an artifact; the Makefile refuses the 32-bit cl.exe before
  nvcc runs (#1405).
• #1436: CI builds against musl (Alpine) and compiles the environment
  tests on Windows, the two holes that let #1430 and #1420 ship.
• #1360 (@kreuzzelg): tier invariants on the fake backend, and the
  reservation leak they found. #1419, #1415, #1418
  (@dajiaohuang): the V4 and fp8 test harnesses build on Windows.
• #1439 (@bherald): the planner tests no longer materialise four 3 GB
  zero-filled payloads. #1358 (@monotophic): a raw-evidence adapter for
  the scoring corpus. #1238 (@ZacharyZcR): reproducible performance
  records are validated.

Docs

• #1446: the qwen36 --ram section said the engine does not stream at
  all. #1392: Azure GLM-5.2 benchmark rows (#1379, #1380, #1384), the rANS
  cross-reference (#1273), and the int8 probe pinned in CI (#1331).
• #1427 (@ZH1995): README.zh-CN formatting. #1438 (@iiEliJas): the
  malloc_trim comment.

colibri 1.10.2

Patch release. Three of these fixes answer reports made against 1.10.1 in the days after it shipped. No new engine; one behaviour change, opt-in.

If you hit one of these on 1.10.1, this is the fix

coli convert produced a GLM-5.3-Flash container the engine refused (#1368). It ran GLM-5.2's converter on everything; Flash nests its language model under the vision wrapper, so the embedding matched no rule, fell through to the generic fallback, and was quantized. The engine was right to refuse it, hours later, inside coli web. coli convert now reads the checkpoint's config.json and picks the converter from it. GLM-5.2's path is unchanged. An option the target converter does not take is refused, not silently dropped (#1369). And tools/convert_glm53.py, the right tool for Flash, is in the archive for the first time (#1364).

coli doctor said two core tensors were missing from a healthy GLM-5.3-Flash (#1365). The check held three literal tensor names taken from GLM-5.2. It now matches roles, prefix-agnostically, so it holds for any family that nests, including ones not written yet (#1366). Nothing was ever missing from those downloads.

The archive was missing files coli reaches (#1359): iq3_pack.py and its grid, reachable only through a subprocess-launched script. The packer follows that edge now, and data files next to a module (#1364).

Also fixed

• RSS guard counts the memory the process owns, not page cache the kernel can reclaim, so a mapped container no longer evicts experts to free memory it was not using (#1350)

• serve honors CANCEL mid-turn (#1336); the disconnect scenario is built, not hoped for (#1329)

• Metal: bit-exact fp8-e4m3 decode (#1346)

• Qwen3.6: tokenizer merges in both spellings (#1319)

• macOS: Homebrew prefixes found when brew is off the PATH (#1320); DeepSeek V4 reports real CPU and memory on macOS (#1308)

• doctor omits the GPU plan for a CPU-only engine (#1322); a missing core tensor explains itself on every fatal path (#1318)

• GLM-5.3-Flash --no-think is the template's lowest effort level, not a prompt shape of ours (#1327)

• The converter refuses a checkpoint it cannot serve, with a per-family pointer (#1305)

• GLM-5.3-Flash serve can use the 16 KV slots the engine has; the registry declared 1 (#1283)

• Security: image_url local reads refuse .., and with COLI_IMAGE_ROOT set the path must resolve inside it, so an authenticated client of a non-loopback server cannot read arbitrary files through the image API; errors no longer confirm a path exists (#1354, dmoraesrs). CI runs with contents: read and third-party actions are pinned by SHA

Supported checkpoints, stated plainly

• GLM-5.3 (744B-class) runs on the colibri engine. It is the same base model as GLM-5.2 (Z.ai say so; the configs differ only by the transformers version that wrote them), so the family is now named GLM-5.2/5.3 rather than announcing one as the other (#1367).
• DeepSeek V4 Flash REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b, 85 GB, 132 of 256 experts) loads with the same engine and no conversion (#1310), confirmed by a second user. It is documented, and the banner reports it by its measured geometry, 43L x 132E, not as 284B (#1371).

Changed

• qwen36 VRAM tier promotes int8 experts instead of reserving for nothing (#1334)
• ...and the three tier bugs that surfaced with it are fixed: an is_x overrun with two or more GPUs, a shutdown that could hang, and a use-after-free on int8 containers (#1339, #1340, #1341; fix #1344 by crichalchemist)
• Kimi K3 stops loading a DSA indexer nothing reads (#1335)
• Opt-in COLI_MAP_EXPERTS=1: experts served through a per-shard file mapping (#1325). Off by default; output byte-identical either way

Docs

• Windows DeepSeek V4 users are led to the release launcher, coli.cmd, instead of a source build (#1291)

Build

• Makefile lists the headers each engine includes as prerequisites (#1284, #1349)
• Site and READMEs carry the eight families with real RAM figures under a contract test (#1302)

colibri v1.10.1

Packaging repair for the prebuilt archives. No engine changes: if you build from source, v1.10.1 is v1.10.0.

What was wrong

The v1.10.0 prebuilt archives were missing seven Python files, which broke four coli commands (#1296, thanks to simongonzalezdc for the report):

• coli convert, coli eval, coli mirror and the benchmark fetch died with "No such file or directory"
• images sent to Qwen3.8 through the server failed claiming Pillow and numpy were missing, when the real problem was a file we had not shipped

What changed

The release workflow no longer lists the Python files by hand. A packing script computes everything coli reaches, through imports and subprocess calls alike, and the same computation runs a second time as a check on the extracted archive before publishing. The old check only confirmed the one file the copy step had just written, which is why v1.10.0 went out green.

Each archive now carries 14 Python files, verified by opening the archives rather than trusting the build log.

Also in this tag

• Engines now reject numeric arguments they can only read half of: ./qwen38 3x2 errors out instead of silently running with a cache of 3 instead of 32 and looking slow.
• The CI Python job now builds the engines its tests need, so seven tests that had been silently skipped run again.

Upgrading

Replace the extracted directory with the new archive. Models, containers and flags are untouched.

colibri v1.10.0

Qwen3.8-Flash-Next

The eighth family, and the first that arrives complete: engine, tool calling and vision together, rather than as follow-ups.

125B ordinary plus 51B hashed n-gram. 48 layers, as twelve blocks of three Gated DeltaNet and one QSA. 512 experts, top-10, plus a shared one. Resident 9.2 GiB; with cap 32, 16.5 GiB in total.

SNAP=/path/to/Qwen3.8-Flash-Next-FP8 ./qwen38 32
./coli chat --model /path/to/Qwen3.8-Flash-Next-FP8

The engines take the snapshot directory in SNAP and the per-layer cache as a positional argument. coli is the launcher and takes flags.

Qwen3.8 is not converted: there is no int4 container for this family and no converter. Download the official FP8 checkpoint (Qwen/Qwen3.8-Flash-Next-FP8, 185.5 GB) and run it directly; the engine reads FP8 natively and quantizes experts into its RAM cache at load time. An earlier revision of these notes named the directory qwen38_flash_next_i4, which wrongly implied a converted container -- if that sent you looking for a conversion step, see #1304.

Tool calling. Qwen3.8 does not use the JSON tool dialect the other families speak. Its format is closer to XML:

<tool_call>
<function=get_weather>
<parameter=city>
Rome
</parameter>
</function>
</tool_call>

Handled end to end through the OpenAI-compatible server, so existing clients work unchanged.

Vision. Images work through the same server. Preprocessing is pinned against the official Qwen2-VL processor, including dynamic resolution: a 1080p frame becomes 2040 tokens. The tower is 27 blocks with learned positions bilinearly interpolated and 2D RoPE on top, verified token by token against the upstream reference.

Fixes

• Qwen3.6 int4 expert unpack is now vectorised. 2.10x faster CPU decode, bit-exact output. Thanks to @lornecodes.
• Qwen3.6 buffer widths and size_t casts corrected. Thanks to @Petsku01.
• Mirror stripe chunks are sized by measured bandwidth instead of split evenly, so a slow leg no longer holds back the fast ones. Thanks to @Unknown-Findout.
• A reasoning family was given the same output budget as a non-reasoning one, so on GLM-5.3, Inkling and Kimi the reasoning could consume the whole allowance before the answer began. The interactive budget is now 16384.
• The pilot prefetch worker could outlive the model it was reading from.
• A malformed container whose two config files disagree could overflow the heap in load_meta.

Those last two were found because make clean was not removing two engines, which meant the sanitiser job had been re-running an old binary with no instrumentation in it. The job was green and empty. Both bugs predate this release.

Benchmarks

Twelve community hardware datapoints are now in the benchmark tables.

Known issues

#1278: GLM-5.3-Flash chat can end on thinking with --no-think. Open, cause not yet identified, not a regression in this release. If you hit it, that issue is the place to say so.

Upgrading

Nothing to change. Existing models, containers and flags keep working.

colibrì v1.9.0

colibrì v1.9.0

GLM-5.3-Flash, and it sees

A seventh family: GLM-5.3-Flash, 321B parameters across 45 layers plus MTP, with a vision tower. It runs on 25 GB of RAM with its routed experts streamed from disk, and it reads images.

prompt:  "trascrivi esattamente il testo che vedi" + a PNG
answer:  COLIBRI SUPPORTA VISION CON GLM 5.3

That is the real 195 GB checkpoint on a laptop, transcribing a banner exactly, both lines. Asked to describe a red square on black it answers Un quadrato rosso su sfondo nero — in Italian, because the question was.

Every mechanism was validated in isolation against a transformers oracle before assembly, which is why the first full forward was token-exact rather than nearly right: KDA, the k-pooled DSA indexer, mHC shared with DeepSeek V4, the ViT, then the whole model, then the whole model with an image.

MLA is absorbed. Caching expanded keys costs 1.39 MB per token across the DSA layers, which is 11.9 GB at 8192 positions on an engine that exists to fit in small memory. Folding kv_b_proj into the two ends leaves the 512-wide latent in cache instead: same arithmetic, an identity rather than an approximation, 33 KB per token.

Where the speed stops, measured

resident weights at GLM53_BITS=4	~12 GB
KV state	33 KB per token (1.1 GB at 32k)
prefill workspace	flat, ~60 MB at any context
decode	~44 s/token cold, ~20 s/token with a warm cache

One token touches 42 sparse layers × 8 experts × 14.2 MB = 4.8 GB. Measured with O_DIRECT, the reference disk gives 72 MB/s at queue depth 1 and saturates near 200 MB/s from QD4. That puts a floor of 24 seconds per token on that hardware: what the disk takes to deliver the bytes, with any CPU and any GPU. Faster silicon does not move it; fewer bytes would. Anyone benchmarking this should know it before starting rather than after.

Vision that is reachable

Paste a path in coli chat, attach or drop a file in coli web, or send an OpenAI image_url part. Windows paths work from WSL, and when an image fails to attach the turn is refused rather than sent — silence there means waiting minutes for an answer about nothing.

Preprocessing is pinned against the official processor: identical geometry on every shape, bit-identical pixels wherever no resampling happens, 0.03 worst case where it does. Images travel in their own IMAGE frame (docs/serve_protocol.md). GLM53_MAX_IMAGE_TOKENS caps how large a picture may become — the checkpoint's own ceiling of 8000 is 2691 tokens for an ordinary 1080p photo.

Tool calling is complete and pinned byte for byte against chat_template.jinja. Reasoning can be switched off (--no-think) or dialled (--effort); both are time controls on this engine, not matters of taste.

tools/convert_glm53.py downloads and converts in one pass, one shard at a time, so peak disk is the output plus a single 5 GB shard rather than the repository's 328 GB. 62 shards, 194.7 GB out, 25 hours.

Distributed inference: Segment and Edge

An engine-neutral runtime ABI, with range-native adapters for all seven engines and a conformance gate that is all-or-nothing across the family set. Segment runs a range of layers on activations; Edge is the two ends, turning tokens into state and state back into tokens, now including full final-head logits so sampling policy stays with the caller instead of inside model math.

Fixes and improvements

Windows. The release archive now has something to double-click (#1241), and the check suite got the timeout it had outgrown rather than being cancelled at the line.

CUDA. Speculative verification pins its expert kernel family so it cannot switch into row-count-gated Tensor Core paths mid-verification; the default path is unchanged. NVCC_ALLOW_UNSUPPORTED makes a host compiler newer than VS2022 reachable. Qwen3.6's CUDA tier decays heat periodically, so a long session can replace an old workload's hot set instead of freezing it, and its admission threshold no longer overflows a saturated counter.

Correctness. Qwen3.6 escapes literal quotes in OpenAI output, which was producing invalid JSON. Vulkan expert residency is reported truthfully in TIERS, HWINFO and EMAP. Two hardcoded /tmp paths are gone.

Planner. It emits evidence-backed next actions instead of silently enabling optimizations it has not measured, and it counts dense weights as they will sit in RAM rather than as they sit on disk — a family that requantizes at load was being told it had no room for a single expert slot per layer when seventeen fit.

Performance. The DeepSeek V4 indexer no longer allocates per batch, and a hot SIMD loop drops a libm call that clang would not inline.

Docs. A Mintlify site with an OpenAPI reference, docs/glm53-flash.md, the GLM53_ environment table, and the IMAGE frame written into the protocol specification.

Thanks

@ZacharyZcR, @Unknown-Findout, @hamedrabah, @bherald, @iam-saiteja, @TechPizzaDev, @Petsku01 and everyone who filed an issue this cycle. The GLM-5.3 oracle contracts are @ZacharyZcR's, delivered as the infrastructure half of a larger proposal after review — and regenerating that fixture with his tool produces byte-identical results to the one this work was developed against, which is worth more than either oracle alone.

colibri v1.8.0

60 pull requests since v1.7.0. The cycle's spine: every engine now finds a
resident expert in one probe instead of a scan, prefill stopped re-reading
bytes it already had, and the DeepSeek V4 CUDA floor dropped from Ampere to
cards from 2016. Everything risky is opt-in; every default path is covered by
token-exact oracles.

Expert lookups: O(1) on every family

• #1164, #1174 — DeepSeek V4 resident-expert lookup made O(1), and miss
  cost made independent of the configured cache size.
• #1193, #1215 — the same expert-to-slot index extended to Inkling and
  Kimi K3, then to GLM, OLMoE and Qwen3.6: a resident hit is 1 probe where
  the legacy worst case was 44 to 219, validated by model-free tests with a
  deterministic probe counter. Every lookup revalidates the slot identity, so
  stale bookkeeping can only become a miss, never wrong weights.
• #1180 — V4 prefill reuses the expert cache across layers and batches
  shared-expert work; the LRU used to re-read the same experts several times
  per layer under prefill's expert-major sweeps.
• #1162, #1166, #1171, #1178 (@TheOptik) — duplicate concurrent expert
  loads coalesced; FP4 expert matmuls batched for prefill and specialized per
  phase; fp8 attention weights decoded with SIMD instead of per-element
  gathers.
• #1197, #1200 — Inkling and Qwen3.6 batch dense and shared-expert prefill.

DeepSeek V4: a wider GPU tier

• #1181 (@8PotatoChip8) — CUDA_ARCH=portable-pre-ampere NO_TC=1 brings
  the whole CUDA tier to Pascal and Turing (GTX 10 / RTX 20 series); the
  loader floor drops accordingly.
• #1202, #1203 — opt-in DSV4_HYBRID=1: on a VRAM-tier miss the missing
  experts are split between the GPU fill branch and the CPU branch, sized by
  bandwidths measured at runtime, with the fill DMA overlapped against host
  compute and host slabs pinned on Linux.
• #1213 — opt-in COLI_CUDA_MOE_DOUBLE=1: a second full-layer expert
  bank is prefetched on a dedicated stream while the current layer computes,
  and the banks swap at the layer switch. Falls back to single-bank when VRAM
  is short.
• #1211 — the shard index is built once per engine open instead of six
  times, the config parsed once instead of three, and the fp8 row transpose
  vectorized bit-identically; a v4_open stderr line reports the split.
• #1141 (@dcutugno) — measured guidance for V4_LOADER_LANES with and
  without the GPU tier.

Both DSV4_HYBRID and COLI_CUDA_MOE_DOUBLE are experimental: defaults are
byte-identical with the envs unset, and they are looking for community
numbers on real cards.

Kimi K3: checkpoints, a vendor oracle, tool calling

• #1188 — a tiny independent oracle validates the engine's KDA and MLA
  paths token-exact against Moonshot's reference implementation, in CI,
  without torch or network.
• #1192, #1194 — opt-in recurrent-state checkpoints (COLI_K3_CKPT=N,
  disk-parked via COLI_K3_CKPT_DIR): an edited or follow-up prompt restores
  the deepest surviving checkpoint and re-prefills only the tail instead of
  replaying the whole conversation through the SSM layers.
• #1208 — K3_VK_UP=auto sizes the Vulkan expert-tier upload from
  measured bandwidth instead of a fixed guess.
• #1144 (@ZacharyZcR) — tool calling wired up with XTML rendering per
  Moonshot's reference renderer.

KV cache quantization (@ZacharyZcR)

• #1151 — CPU KV cache quantization lands: KV8 (fp8 e4m3) and KV_TQ.
• #1155, #1207 — the flash and gather kvb arms became KV8/KV_TQ aware,
  and KV8_GS adds grouped latent scales following FlashMLA's fp8 cache
  geometry.
• #1142 — prefill k/v reconstruction tiled past a size ceiling
  (cause 2 of #768).

coli tune: measured, for every engine

• #1196 — tune measures every engine through the serve protocol, not
  just GLM.
• #1209 — tune measures safe RAM and cache caps on the machine instead
  of deriving them from totals.
• #1195, #1201, #1175 — cap resolved from the plan when --cap is
  absent; rotating workloads persisted and V4 loaders tuned; Metal reported
  correctly in the plan label on Apple Silicon.
• #1198 (@Mr-Neutr0n) — an explicit RAM_GB is clamped to the memory
  actually available (#759).

Fixes across the fleet

• #1220, #1222 — since the V4 CUDA tier landed, the launcher's --gpu and
  --vram check only knew the Windows DLL and refused every valid Linux
  make deepseek-v4 CUDA=1 build (#1219, a 1.7.0 regression); detection now
  reads the binary's actual CUDA linkage, rejects lookalikes, and names the
  right build per platform. Reported by @ANBAL534.
• #1134 (@tonnthuir) — RSS guard measurement and reclamation fixed under
  Linux.
• #1167 — io_uring expert reads now honor the mirror replica (#1165).
• #1152, #1153, #1156 — Inkling Metal MoE qgs at the callsites; the web
  UI renders the reasoning stream instead of dropping it (#1148).
• #1186, #1187 (@waizuichougou) — Qwen3.6 discovers visible CUDA devices
  for the default tier and skips the discarded final decode.
• #1170 (@jerome-benoit) — make install builds qwen36 first.
• #1133 (@QQSHI13) — compress_ratios longer than num_hidden_layers
  accepted.
• #1185 — the GLM banner labels bit widths as compute, not storage.

Tools, bench, CI, docs

• #1105, #1107 (@monotophic) — full-family FP8 e4m3 container mint with
  content-validated resume; a per-token logprob channel, echo prefill
  read-out, and token-ID prompt intake.
• #1205, #1206, #1131, #1132, #1138, #1145 (@ZacharyZcR) — per-record
  layout census for rANS containers; datapoint fixes for Windows 11 labeling
  and win32 RAM probing; the tok/s bound split out of the determinism test;
  V4 docs cross-references.
• #1139 (@SyedYousufFaizan) — datapoint.py parses DeepSeek V4 logs.
• #1182 (@Unknown-Findout) — datapoint checks temp free space before the
  eviction write.
• #1184, #1161, #1127 (@view321) — datapoint survives slow engine
  teardown; persistent rotating-prompt bench datapoints; iobench sub-MB
  blocks and a Windows [threads] fix.
• #1160 — macOS CI compiles every Metal-capable engine with METAL=1.
• #1216 — the README caught up with what already runs: the V4 CUDA floor,
  the six families, K3 checkpoints, and the experimental V4 GPU levers.