Three MoE families now run on the same engine, from 744B to 2.8T parameters — and a 975B model answers on a 25 GB machine.
Pure C, zero engine dependencies, storage/RAM/VRAM treated as one inference hierarchy.
New: two more model families
Kimi K3 — 2.8T total / 104B active (#676)
A sibling engine (c/kimi_k3.c) that streams Moonshot's QAT-trained MXFP4 routed experts straight from the original Hugging Face shards — never re-encoded, never converted — and quantizes the bf16 dense set at load time. Includes the KDA + gated-NoPE-MLA + AttnRes + LatentMoE architecture, an MXFP4 matmul kernel (scalar, AVX2, and int8-activation), chunked prefill (bit-identical, 2.6×), parallel O_DIRECT expert reads, and --chat for K3's XTML format.
Preview status, stated plainly: the engine and tokenizer are validated — the multi-turn wire matches Moonshot's official
encoding_k3.pyat 77/77 token ids — and the regression suite passes. Full-model generation is still gated on a host with ~1.6 TB of storage for the snapshot; nobody has run it end to end yet. If you have that hardware, this one is yours to try.
Inkling — 975B total / 41B active, now on a small host (#701)
Two things stood in the way, and both are fixed:
The dense weights didn't fit. The pre-converted container ships int4 experts but bf16 dense weights — 49.4 GB resident, needed for every token, and expanded to f32 while loading (~99 GB peak). A new one-pass tool quantizes just the dense set to int4-gs64 (15.3 GB). The engine auto-detects it; without it the bf16 path is untouched.
The expert cache silently computed with the wrong experts. moe() acquired a cache slot for every (token, expert) pair up front and held the pointers until compute — but a full cache evicts slots already handed out in the same call. The pointer stays valid; the slot now holds a different expert. No crash, no warning, just incoherent output. Correctness therefore required a cache big enough for every distinct expert in the batch: up to 108 slots per layer for an 18-token prefill at topk=6. Now the experts are processed in rounds and accumulated, so nothing is evicted while in use — and the cache can go down to one slot per layer.
The result, on a 25 GB box:
== Inkling C engine, 66 layers, experts @ container, cache 2/layer ==
<|message_model|><|content_text|>My name is Inkling.<|end_message|>
[eos after 8 tokens] RSS 15.1 GB
Verified against the transformers oracle at 36/36 teacher-forced and 24/24 tokens, at both cap=4 and cap=1 (where the old code corrupted).
One command line for all three
coli reads the model's config.json, picks the engine binary and renders that family's chat template. Nothing about the invocation changes between models:
COLI_MODEL=/nvme/glm52_i4 ./coli chat
COLI_MODEL=/nvme/inkling_i4 ./coli chat
COLI_MODEL=/nvme/kimi_k3 ./coli chat
./coli web --model /nvme/inkling_i4 # API + dashboardFor the non-GLM engines coli chat starts the gateway locally and attaches the TUI, so TUI, API and dashboard share one arch-aware template.
Correctness
- #692 —
attention_absorb_ragged_kernelapplied per-row scales to group-scaled (fmt=4) tensors, so the recommended g64 GLM container produced gibberish throughcoli serve/coli webwhilecoli runwas fine. Located to a single line, fixed with the sharedabsorb_scale()helper, with a two-group regression fixture and the previously-orphaned ragged test wired into CI. Field-confirmed fixed by the reporter on his exact reproduction. - #679 — serve honours
STOP/CANCELmid-turn in single-slot speculative decode. Before, a stop matched by the gateway sat unread untilmax_tokens: on a CPU host that reads as a hang. - #681 — HIP builds work on AMD GPUs without matrix cores (MI50/Vega, RDNA1/2), plus a latent bug where the tensor-core INT4 branch could launch an empty kernel and keep stale output.
- #659 — OOM on integrated/unified-memory GPUs (GB10 / DGX Spark): the RAM budget was computed before the VRAM expert tier was placed, and on unified memory both draw from the same pool.
- #665 —
doctorandresource_planrecognize AMD/ROCm; a working HIP build is no longer reported CPU-only.
Performance
- #691 —
CUDA_RELEASE_HOSTnow defaults on for single-GPU hosts with a largePIN_GB, where the duplicated host copy was starving the RAM expert tier. Measured on a single H200: 1.10 → 4.75 tok/s (3.8×), 9,297 → 14,951 resident experts. - #704 —
TOPPfor Inkling (GLM and Kimi already had it): adaptive routed-expert trimming, where every dropped expert is ~28 MB not read on a disk-bound host. It reports what it trimmed ([topp] 0.30: 36/72 routed used, 50.0% trimmed) — a quality lever you can A/B, not a claim to believe. Off by default, andTOPP=0is bit-identical. - #694 — one loud line when resident tensors fall back to CPU, so
CUDA_EXPERT_GB=autofilling the card no longer degrades 60× in silence. - #698 —
olmoestopped callingfadvise(DONTNEED)after every expert read, which dropped pages the LRU immediately wanted again. - #690 —
COLI_DRAFT_CORPUS: a frozen-corpus draft source proposing whole spans (+22% wall clock when replaying a prompt already in the corpus; the docs state the scope, not the headline).
Thanks
Most of this release is other people's work, and the hard parts were found by people with hardware we don't have:
- @steve-m — the Kimi K3 engine, and for absorbing a parallel PR cleanly instead of duplicating it
- @ZacharyZcR — the Kimi chat/API/Web integration, and for locating #692 to a single line while three of us were chasing the wrong theory
- @terrizoaguimor — four measured reports with A/Bs and medians, three PRs, and the field confirmation that closed #685
- @MrMetric, @ariannamethod, @monotophic, @tasoo-oos, @RDouglasSharp, @khalilswdp
- @remip2, @kvbkvbkvb, @mrchevyceleb, @snailwei, @AlphaMo99, @sepotvin — for reports precise enough to debug from
Verify your download against SHA256SUMS.txt. Full suite green: 28 C tests + 233 Python tests.
