Repository navigation
KrillLM v0.4.0
Headline: KrillLM is now a fully native Swift+MLX inference stack with no
Python dependency. Every mixture-of-experts family (Mixtral, Qwen2-MoE,
OLMoE, DeepSeek-V2, alongside the already-native Qwen3 and Gemma 4
26B-A4B) runs natively; grammar-constrained decoding (JSON, JSON-schema,
regex, and full CFG) and continuous batched serving land; the embedding
stack grows to ~15 encoder families; and the default port flips to
11434 for a zero-config Ollama drop-in. KrillLM beats Ollama on every
published Gemma 4 SKU (decode, prefill, and total wall time).
Added
-
Native MoE runtimes for the remaining mixture-of-experts families.
Mixtral (MixtralForCausalLM), Qwen2-MoE (Qwen2MoeForCausalLM), OLMoE
(OlmoeForCausalLM), and DeepSeek-V2 / V2-Lite (DeepseekV2ForCausalLM, MLA
attention + YaRN RoPE) now run on the native Swift+MLX engine, joining the
already-native Qwen 3 MoE. Each uses thegatherQuantizedMMSwitchGLU expert
dispatch and the shared prefill sort path; each is verified for logit parity
against mlx-lm on a tiny quantized checkpoint (argmax + cosine > 0.9999).
Qwen2-MoE adds a sigmoid-gated shared expert; OLMoE adds whole-projection
q/k-norm attention; DeepSeek adds MLA, YaRN, an always-on shared expert, the
first_k_dense_replacedense prefix, and V2/V3 group gating. DeepSeek-V3's
absorbed-MLA checkpoint layout (and the RAM-blocked 671B model) is deferred
with a clear load-time message (seedocs/BACKLOG.md). -
CFG-constrained decoding (Stage D) (follow-up #9): a request can now
constrain the output to a full parse of a context-free grammar, for shapes
with unbounded balanced nesting that regex cannot express (recursive
expression languages, nested blocks, custom DSLs). Supplied via an OpenAI
response_format: {type:"lark", grammar:"<grammar>"}(ortype:"cfg")
extension, or an Ollamaformat: {"lark":"<grammar>"}/{"cfg":"<grammar>"}
object. The grammar dialect is regex-flavored with named rules: rules
name: body, alternation|, groups(), quantifiers? * +, string
literals"...", single-character classes[...], and bare identifiers as
nonterminal references (which is what enables recursion); the start symbol is
the rule namedstartif present, else the first rule. A newCFGGrammar
compiles this to pure BNF (string literals, groups, and quantifiers desugar
to anonymous nonterminals over single-character terminals reused from
RegexGrammar) and drives the sharedGrammarTokenMaskwith a
character-level Earley recognizer, which correctly handles left-recursion,
ambiguity, and nullable rules. The grammar is matched as a full parse: EOS is
allowed only when the start symbol spans the whole output. An uncompilable
grammar (syntax error, undefined nonterminal, oversized) disables the mask and
the request decodes unconstrained with the system-prompt guidance; CFG output
is not JSON, socoercereturns it verbatim. Opt-in / slower: an Earley
chart is near-unique per prefix, so the per-state mask cache mostly misses and
each decoded token pays a full-vocab rescan - suited to short, structurally
constrained outputs. This completes the grammar follow-up (Stages A-D). -
Regex-constrained decoding (Stage C) (follow-up #9): a request can
now constrain the output to a full match of a regular expression, for
non-JSON shapes like enums, dates, IDs, and phone numbers. Supplied via
an OpenAIresponse_format: {type:"regex", regex:"<pattern>"}(or
type:"grammar") extension, or an Ollamaformat: {"regex":"<pattern>"}
object. A newRegexGrammarparses a bounded regex dialect (literals,
., character classes incl. ranges and negation, the\d \w \sfamily
and escaped metacharacters, groups, alternation, and the* + ? {n} {n,} {n,m}quantifiers) into a Thompson NFA and drives the shared
GrammarTokenMaskby subset construction, so aStateis a compact set
of NFA nodes that recurs across generations and caches well (no
per-prefix blow-up). The pattern is matched as a full match: EOS is
allowed only at an accepting state. An uncompilable or unsupported
pattern (backreferences, lookaround, counted repetition on a group)
disables the mask and the request decodes unconstrained with the
system-prompt guidance. Regex output is not JSON, socoercereturns it
verbatim rather than JSON-extracting. A full Lark-style CFG runtime
(arbitrary nesting via an Earley parser) remains a possible follow-up;
regex covers the common flat-pattern cases, and JSON/JSON-schema (Stages
A/B) already cover the common nested cases. -
Grammar-constrained JSON decoding (Stage A) (follow-up #9):
format:"json"/ OpenAIresponse_formatnow drive a real token-level
logit mask, not just guided prompting. A newKLMGrammarmodule runs an
incremental JSON-value automaton (JSONGrammar) plus a tokenizer-vocab
mask (JSONTokenMask) so the sampler can only pick tokens that keep the
output a valid JSON prefix; EOS is forbidden until the value is complete.
This turns JSON output from "usually valid" into "always valid",
including under adversarial prompts. The mask is built once per model and
cached per grammar state.Samplergains an optionalmask:parameter
applied before greedy/temperature/top-p/top-k/min-p, so the decode path
is byte-for-byte unchanged when no format is requested. The existing
system-prompt injection + post-extractioncoerceare kept as a fallback
(and remain the schema-shape enforcement for.schemarequests until the
Stage B schema→grammar compiler lands). Constrained requests take the
serial decode path — the mask advances a per-sequence automaton, which
the batched/speculative loops cannot interleave. -
JSON-schema constrained decoding (Stage B) (follow-up #9): a
formatcarrying a JSON Schema (OpenAIresponse_format: json_schema
/ Ollama objectformat) now constrains decoding to the schema's
structure, not just JSON well-formedness. A newSchemaGrammar
automaton plus a totalSchemaGrammar.compile(_:)turn a bounded
schema subset into a pushdown grammar: objectproperties/required
/additionalProperties(bool or sub-schema), arrayitems, scalar
type(string, number, integer-distinct-from-number, boolean, null),
andenum/const. The Stage A mask layer was generalized to a
GrammarAutomatonprotocol +GrammarTokenMask<A>, so JSON and schema
masks share all caching, fail-open, and per-generation session logic
(JSONTokenMaskis now a typealias). Unsupported keywords (anyOf,
oneOf,allOf,not,$ref,patternProperties,pattern,
format, numeric / item bounds, uniontypearrays) relax to an
unconstrained value (still valid JSON) with a one-time note; an
uncompilable schema falls back to the Stage A JSON-validity mask, and
the system-prompt guidance + post-extractioncoerceremain the
backstop. Stage C (general Lark-style grammars) is the remaining
follow-up. -
Continuous batched decode (follow-up #8, Stage C1): batching is now
continuous rather than a static cohort. A persistent per-model
ContinuousBatcherruns one decode loop that admits a newly-arrived request
into the running batch at the next step boundary and drops finished or
cancelled rows mid-flight (shrink), instead of waiting for a fixed cohort to
complete. Each row keeps its own authoritative[KVCache]; the active set is
re-stacked left-padded only when it changes (an "epoch"), so within an epoch
the per-step cost matches the verified Stage B core and every active row is
always decoded exactly as if it ran alone. Driven via a new
InferenceEngine.submitBatched(_:maxRows:windowMs:); theBatchSchedulernow
forwards eligible requests to it (rolling admission) and the static
generateBatchedcohort entry is retained for direct/test use. Scope is
unchanged from Stage B: text-only Llama 3.x / Qwen 2.5-3 dense, fp16 KV, no
prefix cache, no speculative decode (Gemma 4 / MoE / VL and
prefix/int8/spec reconciliation are later Stage C sub-stages). The batcher is
torn down onunload(), finishing every in-flight and waiting stream so a
model swap never strands a request. -
Live batched serving (
BatchScheduler) (follow-up #8, Stage B —
wiring): concurrent same-model requests are now coalesced into ONE batched
forward, turning the verified Stage B engine into a live throughput feature.
A per-modelBatchSchedulergathers eligible requests into a static cohort
(up toKRILL_NUM_PARALLEL, within a smallKRILL_BATCH_WINDOW_MSwindow,
default 8 ms) and drives a new streamingInferenceEngine.generateBatched
entry that demuxes each row to its own token stream — with per-row sampling
(temperature / top-p / top-k / penalties), per-row stop/maxTokens, and
ragged prefill.KRILL_NUM_PARALLEL < 2, ineligible families, multimodal
requests, seeded non-greedy sampling, and explicit speculative opt-ins fall
through to today's serial path byte-identically. fp16 KV only; the
prefix cache and speculative decode stay bypassed on the batched path;
finished rows remain in the batch until the cohort completes (mid-flight
shrinking and continuous admission are Stage C). -
Batched concurrent decode engine (follow-up #8, Stage B - core): the
inference engine can now decode several ragged-length prompts in ONE
batched forward for plain-causal families (Llama 3.x, Qwen 2.5/3 dense).
Each row carries its own RoPE position (threaded into the attention
forward) and an additive mask hides its left-padded prefix in the stacked
KV cache, so a batched row reproduces that prompt's solo decode. Verified
on real checkpoints: batched per-row logits match the single-prompt logits
within fp16 rounding (~1 ULP), with no cross-row attention bleed. fp16 KV
only; greedy/per-row sampling; speculative decode and the prefix cache are
bypassed on the batched path. This lands the verified engine; wiring it to
concurrent server requests (KRILL_NUM_PARALLEL >= 2) is the next PR. -
Multi-model resident pool (
MAX_LOADED_MODELS > 1) (follow-up #8,
Stage A — routing first): a newEngineRegistrykeeps more than one
model resident at once (KRILL_MAX_LOADED_MODELS/
OLLAMA_MAX_LOADED_MODELS, default 1). Generate requests are now
routed-or-loaded by model name: a request for an installed model
loads it on demand and routes to it, keeping previously-loaded models
resident up to the cap instead of discarding the prior model on every
swap. Eviction is in-flight-aware: the least-recently-used resident
that is NOT currently generating is evicted, and when the pool is full
and every model is busy a new-model request gets a meaningful 503
(namingKRILL_MAX_LOADED_MODELS) rather than tearing a model down
mid-stream. All resident engines share one prefix cache (keys already
namespace by model). At the defaultMAX_LOADED_MODELS=1behavior is
unchanged. (Batched concurrent decode is Stage B.) -
Per-model keep-alive (follow-up #8, Stage A-2): each resident model
carries its own idle deadline, so a request'skeep_alive(default /
-1to pin /0to evict-after-drain) applies to that model alone. The
background evictor now unloads each model independently when its own
deadline passes and it is idle, leaving other resident models loaded
(previously a single global deadline unloaded the whole pool).
GET /api/pslists every resident model with its ownexpires_at,
not just the active one. AtMAX_LOADED_MODELS=1this matches the prior
single-model behavior. -
Gemma 4 26B-A4B native text MoE (#81): first native inference of
the sparse 26B-A4B variant on Apple Silicon. Router + top-K expert
dispatch in Swift+MLX. Closes #80. -
KLMAgent skeleton (#65):
OperatorLoop,OperatorTool,
OperatorEvent,HardwareInfo, andRecommenderland the structural
foundation for agent mode (slice 3 sub-PR A). Tool wiring + CLI follow
in later sub-PRs. -
Daemon-mode CLI routing for
krillm run(#63): when akrillm servedaemon is already running,krillm rundetects it (probes
/v1/status), routes the request through/v1/chat/completions, and
skips the per-call model load. TTFT drops from seconds to tens of
milliseconds (~5x warm-daemon speedup). Text-only single-shot
requests are routed; multimodal, draft-model, Modelfile-override, and
REPL paths stay in-process.KRILL_NO_AUTO_DAEMON=1forces
in-process. -
Modelfile
TEMPLATEoverride applied at decode: created models
carrying aTEMPLATEdirective now render their prompt with it
instead of the model's built-in chat template. OllamaTEMPLATEs are
Gotext/template, so this ships a from-scratch Go-template engine
(GoTemplate: actions, pipelines,if/range/with,{{- -}}
trimming, and theeq/and/len/index/slice/printf/... builtin
set) plus theOllamaTemplateContextbridge from chat messages to the
.System/.Messages/.Promptrender context. The override was
already parsed and round-tripped through/api/show; the renderer was
the missing piece. A template that fails to parse/evaluate falls back
to the built-in chat template rather than failing the request. -
SDK usage docs (#64): verified end-to-end snippets for the OpenAI
Python SDK, LangChain, LlamaIndex, and the Anthropic SDK pointing at
the local server.
Removed
- The Python MoE sidecar bridge. With every mixture-of-experts family now
native, the mlx-lm sidecar (tools/moe_bridge.py,MoEEngine, the
PythonSidecarplumbing, thehandleMoEChatserver path, and the SIGINT
teardown handler) is deleted, along with theKRILL_NATIVE_MOE=0opt-out it
backed. Inference is now fully Swift+MLX-native with no Python dependency. MoE
manifests route through the dense engine like any other native causal LM
(ModelAdapter.chatRoutingno longer has amixtureOfExpertscase), and the
.moefamily reportsproductionNative.
Changed
- SwitchGLU MoE dispatch replaces scatter (#82): the per-layer
scatter dispatch (a Swift loop driven by a host read of per-expert
token counts) is gone. A newGemma4SwitchGLU/
Gemma4QuantizedSwitchedLinearruns onegatherQuantizedMMkernel
per (gate, up, down) projection across all top-K experts with zero
host syncs in the layer loop. Decode (N=1) pays no Swift loop. This
flipped 26B-A4B from 9% behind Ollama to 43% ahead on total wall time. - Gemma 4 vision encoder reads
vision_configfrom the checkpoint
(#79): the SigLIP2 tower shapes are parsed from the model's own
config instead of hardcoded, so checkpoints with different vision
dimensions load correctly. - Default server port flipped
11435->11434(#83): KrillLM now
listens on the same port stock Ollama uses, so existing Ollama
clients connect with no configuration. The previous default11435
still works for one release when set explicitly (--port 11435or
KRILL_PORT=11435). This activates the T0-1 flip that
docs/OLLAMA_MAC_PARITY_PLAN.mddeferred until themac_paritygate
went green (18/18, 2026-05-28). - Quantization config requires explicit
group_sizeandbits
(#74): the decode path no longer falls back to silent defaults when a
checkpoint omits quant metadata; it now requires the values be
present, surfacing malformed quant configs instead of guessing. - Qwen3-MoE SwitchGLU dispatch (opt-in via
KRILL_NATIVE_MOE=1):
the native Qwen3-MoE runtime now dispatches the top-K experts with a
singlegatherQuantizedMMper projection (Qwen3SwitchGLU), the same
pattern PR #82 applied to Gemma 4. The stacked
mlp.switch_mlp.{proj}.*checkpoint tensors bind directly (no
per-expert unpacking), and the per-layer host sync that drove the old
scatter dispatch is gone. Decode on Qwen3-Coder-30B-A3B benches 2.7x
faster (24 -> 66 tok/s). (At the time this landed the unsorted gather
still regressed long-prompt prefill, so the path was opt-in; the #87
sort path fixed prefill and #88 then made native the default - see
those entries below.) - SwitchGLU sort path recovers prefill parity (#87): the unsorted
gatherQuantizedMMdispatch (#82, #85) does anM=1matmul per
(token, expert)with experts gathered in router-score order, which
regresses long-prompt prefill. Mirroring mlx-lm'sswitch_layerssort
step, the SwitchGLU now sorts the flattened(token, slot)
assignments by expert id onceindices.size >= 64(prefill) so each
expert's gather slice is contiguous andgather_qmm's sorted-indices
fast path applies, then unsorts the output back to(token, slot)
order. Decode (N=1, below the threshold) stays on the unsorted fast
path, so the #85 decode win is untouched. Measured on a 256-token
prompt: Qwen3-Coder-30B-A3B prefill 229 -> 536 tok/s (+134%) with
decode held (65 tok/s); gemma-4-26b-a4b prefill ~230 -> 494 tok/s
with decode held (~59 tok/s). Applied to bothGemma4SwitchGLUand
Qwen3SwitchGLU; shared helpers inMoESortPath.swift. This unblocks
promoting native Qwen3-MoE to the default. - Native Qwen3-MoE runtime is now the DEFAULT (#88): with #85
decode (2.7x) and #87 prefill parity both landed, the native Swift+MLX
Qwen3-MoE runtime no longer waits behind theKRILL_NATIVE_MOE=1
opt-in. Qwen3-MoE checkpoints now load, serve, and tool-call on the
native path with no env var;KRILL_NATIVE_MOE=0is the opt-out that
forces the legacy mlx-lm bridge for one release. The model loader,
nativeMoEDispatchSupported, and the server MoE routing all default to
native; the still-unmigrated MoE families (Mixtral / Qwen2-MoE / OLMoE
/ DeepSeek-V3) continue to route to the bridge./api/showreports a
checkpoint-awaresupport_tier: a served Qwen3-MoE checkpoint is now
production_native(the newsupportTier(for:at:)resolves it from the
installed config), while the bridge-only members and the family-level
floor staycompatible_fallback. Verified end-to-end on
Qwen3-Coder-30B-A3B: coherent generation and OpenAI tool calling
(finish_reason: tool_calls) on the native default.
Fixed
- Qwen3-MoE coherence (#78): mlx-community ships stacked
switch_mlpexpert weights; KrillLM now unpacks them into per-expert
keys, so Qwen3-Coder-30B-A3B serves coherent text and tool calls
instead of garbage. - Gemma 4 e4b / 26B-A4B crash on load (#72):
layer_typesis now
parsed from the checkpoint config; the previous hardcoded assumption
crashed these variants. - Mixed-precision quant support (#73): per-module
bits/
group_sizeoverrides let checkpoints that quantize different modules
at different precisions load correctly. - External
chat_template.jinjaloading (#77): the tokenizer loads
an external chat template file when present and bypasses a lossy
round-trip that corrupted some templates. - Model puller HF file allowlist (#71): extended to cover newer
tokenizer file conventions so recent HF repos pull completely. - Removed fake
gemma-4-12b/gemma-4-27baliases (#70): these
SKUs do not exist; the aliases pointed at nothing and are gone.
Performance vs Ollama
Median across 5 runs, warmed servers, 128-token generation, on the M4
target (full report + raw JSONs archived with the release). KrillLM
wins decode, prefill, and total wall time on every published Gemma 4
SKU:
| Variant | KrillLM decode | Ollama decode | KrillLM total | Ollama total | Total delta |
|---|---|---|---|---|---|
| e2b (dense, ~2B, 4-bit) | 110.1 tok/s | 88.4 tok/s | 1.18s | 1.65s | +40% |
| e4b (dense, ~4B, 4-bit) | 62.6 tok/s | 55.2 tok/s | 2.07s | 2.53s | +22% |
| 26B-A4B (sparse MoE, 4-bit) | 61.6 tok/s | 49.0 tok/s | 2.11s | 3.02s | +43% |
The 26B-A4B SwitchGLU rewrite (#82) drove the headline gain: decode
41.2 -> 61.6 tok/s (+50%), prefill 3516 -> 5193 tok/s (+48%), total
3.17s -> 2.11s, flipping it from 9% behind Ollama to 43% ahead.