Skip to content

v0.8.0 - native Gemma-4-12B coder (MLX)

Choose a tag to compare

@srvsngh99 srvsngh99 released this 16 Jun 22:55
· 191 commits to main since this release

KrillLM v0.8.0 adds native MLX support for the Gemma-4-12B coder fine-tune (gemma-4-12B-coder-fable5-composer2.5), converted GGUF-free.

  • New model: krillm run gemma-4-12b-coder runs it on the native gemma4_unified runtime (nvfp4, ~25 tok/s on a 24GB Mac). Published at srv-sngh/gemma-4-12B-coder-fable5-composer2.5-nvfp4.
  • GGUF-free conversion: ingested from the upstream NVFP4 safetensors (compressed-tensors), not lossy GGUF k-quants. tools/convert_gemma4_compressed_nvfp4_to_bf16.py decompresses NVFP4 to bf16 (pure numpy, self-checking), then the proven mixed-nvfp4 requant (8-bit attn o_proj + projectors).
  • Reasoning toggle: enable_thinking for the Gemma-4 channel template (KRILL_ENABLE_THINKING) so the coder reasons before answering. Stock Gemma-4 prompting unchanged.
  • Standalone eval harnesses: coding_eval_standalone.py (HumanEval pass@1, optional --think) + agentic_eval_standalone.py.

Indicative numbers (our own harness, single greedy pass, not EvalPlus-leaderboard-comparable): HumanEval pass@1 89.6% reasoning-on / 82.9% off (n=164); 8/8 on an 8-scenario tool-use suite.

Install: brew tap srvsngh99/krillm && brew install krillm (or brew upgrade krillm) - Use: krillm run gemma-4-12b-coder
Asset: krillm-0.8.0-arm64-apple-macos.tar.gz (Apple Silicon). Changelog: [0.8.0].