Repository navigation
colibri 2.2.0: everything since 2.1.0 (2026-10-31). The full list is in CHANGELOG.md.
New models
- Inkling-2-Small (594B, 17B active, Apache 2.0): text, pictures and tools, from unsloth's GGUFs (UD-IQ1_S 171.7 GB to UD-Q4_K_XL 354 GB, pictures through the mmproj) or our int4 g64 converted from the NVFP4 release; CUDA and Vulkan, an MTP drafter (
INK2_MTP=1). - Inkling-2-Mini (26.4B, 3.4B active, Apache 2.0), whose middle layers run three passes: text and pictures from unsloth's GGUFs (UD-IQ2_XXS 8.5 GB to UD-Q4_K_XL 17.6 GB; their root files and the
Mainline_Supportones, mainline llama.cpp'sinkling2_looplayout, alike) or our int4 g64 from the NVFP4 release; Vulkan and CUDA with a decode step in one frame of their dense chains (the routed experts on the card, the attention split over its positions), and the CPU. UD-Q4_K_XL decodes at 40.9-41.1 tok/s with CUDA, 39.2-39.4 with Vulkan and 8.0-8.1 on the CPU, and reads a 570-token prompt at 285-289 tok/s with CUDA and 270-277 with Vulkan. - Qwen3.8-27B (dense): text and images on CUDA, Vulkan and the CPU, its MTP head from a GGUF, and speculative decoding on CUDA with DFlash2.
- Kimi K3.1 Pro: K3's weights with sparse attention past 2048 tokens, on the CPU, Vulkan and a new CUDA chain.
- Qwen-Image-2.1 from GGUFs, base and Turbo: about 10 GB on disk and a 5.9 GB RAM peak, where the official checkpoint is 33 GB.
- DeepSeek V4.1 Flash on Vulkan and CUDA, GLM-5.3 and GLM-5.3-Flash on the CUDA chain, Qwen3.8 with AMD Quark's MXFP4 experts (CPU).
Faster
Measured for this release, 2.1.0 against it, through coli serve on an RTX PRO 4000 Blackwell (24 GB) with an i5-13500 and 62 GB; prompt in tok/s at 1000 tokens, then 300 tokens of decode, two runs each:
| GGUF | backend | prompt | decode |
|---|---|---|---|
| Qwen3.8-Flash-Next UD-IQ2_XS | CUDA | 29 -> 329-421 | 7.7-7.8 -> 55.8-56.6 |
| Qwen3.8-Flash-Next UD-IQ2_XS | Vulkan | 127 -> 135-137 | 34.7 -> 36.3-36.7 |
| Qwen3.8-Flash-Next UD-IQ3_XXS | CUDA | 27-28 -> 478-492 | 10.5-15.9 -> 41.5-41.6 |
| Qwen3.8-Flash-Next UD-IQ3_XXS | Vulkan | 173-174 -> 276 | 30.7-31.8 -> 33.0-33.4 |
| GLM-5.3-Flash UD-IQ2_XXS | CUDA | 4.2-5.3 -> 17.1-17.5 | 1.9-2.3 -> 5.2 |
| GLM-5.3-Flash UD-IQ2_XXS | Vulkan | 16.1-16.2 -> 17.5-17.7 | 4.7 -> 5.5-5.6 |
| DeepSeek V4 Flash UD-IQ1_S | CUDA | 16.3-16.4 -> 18.3-18.6 | 6.1-6.2 -> 6.1 |
| DeepSeek V4 Flash UD-IQ1_S | Vulkan | 19.2-20.3 -> 21.5-22.0 | 5.8-6.0 -> 6.4-6.5 |
| Qwen3.6-35B-A3B UD-Q4_K_M | CUDA | 3297-3436 -> 3278-3419 | 102.1-104.4 -> 104.4-106.2 |
| Qwen3.6-35B-A3B UD-Q4_K_M | Vulkan | 2127-2149 -> 2171-2208 | 91.3-92.5 -> 91.6-93.1 |
On the CPU (Ryzen 7 PRO 8700GE, 8 cores), the GGUF kernels' new forms (#2868): Qwen3.6-35B-A3B UD-IQ2_XXS reads a 500-token prompt at 148-149 tok/s instead of 125-126 and decodes at 25.9-27.0 instead of 21.6-21.9. The int4 container and Qwen3.8-Flash-Next (vq2 experts) are unchanged there.
The int4 container of Qwen3.6-35B-A3B decodes as 2.1.0 did: 110.0-110.6 tok/s on CUDA (2.1.0: 110.1-111.6), 98.9-99.1 on Vulkan; its prompt reads at 2898-3010 tok/s on CUDA (2.1.0: 2731-2947).
The setup picks the model for your machine
coli setup (START-HERE) used to recommend Qwen3.6-35B on every machine from 12 GB to 512 GB of RAM, card or no card. Now it measures the machine and estimates how fast each model decodes there:
- the RAM's bandwidth, two seconds;
- each card's memory and bandwidth, from the CUDA driver;
- the disk's kind.
It recommends the most capable model that decodes smoothly, at 20 tok/s or more, and the menu shows every model's estimate and where it runs (on the card, half on the card and half in RAM, in RAM, or streaming from the disk). Examples:
- 8 GB laptop: a 2-bit Inkling-2-Mini.
- 24 GB card: Qwen3.8-Flash-Next UD-IQ3_XXS, about 48 tok/s estimated and 41.5 measured on an RTX PRO 4000.
- Server with eight H100: MiMo-V2.6 Pro.
The catalog now has GGUFs too: Inkling-2-Mini and Qwen3.8-Flash-Next from unsloth, imported in place after the download.
The chat and the API
coli webopens the chat in the browser, with documents (PDF, Word, OpenDocument, PowerPoint, web pages) dropped into a message; the old dashboard is gone.coli tuiis the chat in a terminal: one C file with no libraries, the conversations, the answer with its reasoning folded, the monitor and the expert map.coli chatkeeps the classic line chat.- A desktop window (
colibri-desktop) shows the chat page in a window of its own, starting and stopping the server for you. - The prefix store: every prompt an engine reads is kept by content, and a later request on the same system prompt or the next turn of a conversation takes it with the logits a cold read gives (Qwen3.6, Qwen3.8, DeepSeek V4.1, GLM-5.3-Flash, Kimi K3, MiMo, Inkling, OLMoE).
tool_choicenaming a function is honored on Qwen3.6, Qwen3.8, MiMo, GLM-5.3 and Mistral-Large-4;/v1/messagestakesstop_sequencesand hascount_tokens.
Platforms
- A linux-aarch64 archive (Raspberry Pi 4 and 5, Graviton, Ampere).
- FreeBSD x86-64 from source, built with the base system's clang.
- The Windows CUDA package carries every engine with a CUDA path: glm53.exe, qwen38.exe, deepseek_v41.exe, mimo.exe, inkling.exe, olmoe.exe, laya.exe and gliner_decide.exe join colibri, qwen36 and kimi_k3.
coli setupbuilds OLMoE, Laya and GLiNER2.5-Decide with CUDA on an NVIDIA card (OLMoE decodes at 316 tok/s on an RTX PRO 4000, against 32.5 on the CPU).- Qwen3.6 on Apple Silicon's GPU (Metal), from source (qwen36-metal.md).
- In a container with a CPU quota or a cpuset, the OpenMP team follows the quota instead of the host's cores (#2914).
Fixed in this release
- DeepSeek V4 Flash from a GGUF starts on CUDA again (2.1.0 served it; the resident experts' auto mode had refused it since).
- Qwen3.8's RAM plan with its routed experts only in a sidecar (0 cache slots with
Q38_MTP=0, too many with the MTP head on). - The grouped int4 kernels' bits on aarch64, where GCC 16 rounded the fused and the single kernels two ways (#2872).
- Qwen3.8's CUDA chain runs the routed experts on the card whenever every one of them is there, whatever the RAM cache's size. With a GGUF that fits the card and
--ram 16, it had run them all on the CPU (#2966). - The Windows CUDA package loads its DLL before the chain looks for it, so the chain is no longer "absent" on an FP8 snapshot (#2966).
- On Windows, a GGUF's reads no longer queue one behind the other: on an RTX 5090 with Qwen3.8-Flash-Next IQ1_M, a 4,479-token prompt's first token came in 2.0-2.1 s instead of 5.4-5.8, and the engine was ready in 12.7 s instead of 21 (#2967).
- The expert map (the chat page's and
coli tui's) shows where Qwen3.8's experts are on a CUDA card: it had shown every one in RAM or on the disk. It now reads the CUDA tier, and the chain's host pool counts as RAM (#2969). - Bounds on what the chat takes: an Office file is unpacked up to 64 MB a part and 256 MB in all, a picture is taken up to 8192 x 8192 pixels, and the chat page and the desktop window open links only as http and https.
For the open reports
- Qwen3.6 on 8 GB cards (#2733, #2734, #2900): the per-row host pool and the memory the KV mirrors give back (#2741), the RAM plan as the engine holds it (#2904).
- A CUDA step that hung on Windows (#2696): the hybrid wait bounded (#2699) and the WDDM flush (#2718).
- A shorter conversation that never reused its prefix with
--kv-slots 1(#2703), in every engine (#2713, #2718). - FP8 and int4-g64 experts on a CPU without FMA (#2715): #2724.
Please tell us on those issues whether this release changes what you saw.