Repository navigation
FAQ
Krafla is a dependency-free C11 inference runtime for Llama-style language models with INT8 (Q8_0) and INT4 (Q4_0) weights. It runs on the CPU only, on x86-64 from SSE2 up to AVX-512 VNNI and on ARM from 32-bit ARMv7 NEON up to i8mm and SVE. You pick the instruction set (ISA) when you compile, and every operator is bound to the kernel written for that ISA, with no runtime dispatch.
The project was called isa-lm until version 0.5.0. Krafla is a volcanic caldera in north Iceland with a geothermal power station, and the name also reads as Kernels Retargeted Across Families of Legacy Architectures.
Yes. The sse2 build needs nothing beyond SSE2, which every x86-64 CPU has
(Athlon 64, Core 2 and later), and the sse41 build adds SSE4.1 (Penryn,
Nehalem and later). Both have hand-written int8 kernels rather than a scalar
fallback. On the comparison machine the SSE2 build decodes
Qwen2.5 0.5B Q4_0 at 25.7 tokens/s on one core and 77.3 on four. That machine
is a modern Xeon running the SSE2 code, so an old CPU will be slower; the
ratio to llama.cpp's SSE2 build on the same machine was 6–7×.
On Qwen2.5 0.5B with the same Q8_0 and Q4_0 formats on the same machines (details):
- Decode is at least as fast with Krafla at every ISA level measured: 1.2–1.5× with AVX-512 VNNI, 1.0–2.3× with AVX2, 2.1–10× on the SSE2 and SSE4.1 builds, and 1.04–1.7× on AArch64.
- Prefill (prompt processing) is faster with Krafla from AVX2 down and on AArch64, except dotprod Q4_0, where llama.cpp leads by 1.10–1.13×. On a CPU with AMX, llama.cpp's AMX kernels prefill 1.03–1.17× faster than Krafla's AVX-512 VNNI build.
-
Quality: Krafla's Q8_0 scores perplexity 13.32–13.35 against 13.34 for
llama.cpp's F16. Its Q4_0 fits each block's scale rather than rounding it
from the block's maximum. Against llama.cpp's Q4_0 on the same tensors:
- Without an importance matrix, the KL divergence from the unquantized model is 6–9% lower on four Qwen models and 5% higher on Llama 3.2 1B.
- With an importance matrix, llama.cpp's own search matches the fit or does better on five of seven models.
- See Q4_0 block scales.
llama.cpp does much more: GPU back ends, K-quant and i-quant formats, and many more architectures (mixture of experts, image input). Krafla covers Q8_0 and Q4_0 on the CPU, for the ISAs listed above.
The ARM builds cover the Raspberry Pi's CPUs by ISA. Raspberry Pi 2 (ARMv7
NEON) takes the neon build of the armv7hf release; Raspberry Pi 3,
Zero 2 W and 4 (Cortex-A53, Cortex-A72) take neon in either the armv7hf
or the aarch64 release, depending on the operating system; Raspberry Pi 5
(Cortex-A76) takes neon_dotprod. The original Raspberry Pi 1 and Zero
(ARMv6) have no NEON. These builds are tested under qemu and natively on
GitHub's Arm runners (Neoverse), and the CPU guard is checked on emulated
Cortex-A53 and Cortex-A76 cores; none of this has run on an actual Raspberry
Pi yet, so there are no Raspberry Pi speed numbers.
Llama 2, Llama 3/3.1/3.2, TinyLlama, SmolLM2, SmolLM3, Mistral, Qwen2/Qwen2.5,
Qwen3, Qwen3.5 and Qwen3.6 (dense, the text model), Gemma 2 and Gemma 3 (the
text model, 1B to 27B), from their Hugging Face
safetensors files, plus llama2.c checkpoints. tools/convert.py converts a
checkpoint to the .mlb format and tools/convert_tokenizer.py its
tokenizer.json. TinyLlama 1.1B, Llama 3.2 1B, Qwen2.5 0.5B, Qwen3 0.6B,
SmolLM2 135M and Gemma 3 1B and 4B have been run end to end; see
Getting a model.
Fifteen models from 135M to 9B parameters (SmolLM2, SmolLM3, TinyLlama,
Qwen2.5, Qwen3 and Qwen3.5) are converted in the
Krafla collection on Hugging Face, in Q4_0 and Q8_0 and, for all but
the two SmolLM2 models, a mixed Q4 file, each with its tokenizer and a model
card with perplexity and speed: see
Converted models on Hugging Face.
Download a .mlb file and tokenizer.mlt next to a release and run it; no
Python is needed. For other models, convert the Hugging Face safetensors with
tools/convert.py.
In a release archive, the ml_run wrapper tries the builds from fastest to
slowest and starts the first one your CPU supports. When you build from
source, make ISA=... picks one (see Choosing the ISA).
A binary run on a CPU that lacks its ISA exits with code 3 and names the
missing feature instead of crashing.
About the size of its .mlb file plus the KV cache. With the default f16
block scales, Qwen2.5 0.5B takes 265 MiB in Q4 and 500 MiB in Q8, Llama 3.2 1B
663 MiB in Q4 and 1252 MiB in Q8, Gemma 3 4B 2083 MiB in Q4. The KV cache is
n_layers × ctx × kv_dim × 8 bytes, where ctx is the context length set
with --ctx (default 4096, or the model's max_seq if smaller); --kv f16
halves it and --kv q8 makes it about 3.8× smaller.
No GPU. The runtime is C11 with no dependencies beyond the C library (and pthreads for the threaded builds); the release binaries are static. Python 3 with numpy is needed only to convert models and tokenizers.
Yes. ml_server serves a model with the OpenAI API (/v1/chat/completions,
/v1/completions and /v1/models, with streaming), so the openai Python
and JavaScript libraries and tools that take a base URL can use it:
./ml_server Qwen2.5-0.5B-Instruct-Q4_0.mlb -z tokenizer.mlt --chat qwen -t 4
# base URL: http://127.0.0.1:8080/v1It answers one request at a time and keeps the KV cache between requests, so the next turn of a conversation runs only its new tokens. See Server.
Yes: the logits are bit-identical for any thread count and any prompt batch
size, and greedy decoding (-T 0, the default) gives the same text every
run. The tests check this, and check every kernel against scalar and
double-precision references and the whole forward pass against a numpy
reference.
Linux on x86-64, AArch64 and ARMv7, and Windows on x86-64 (MinGW release binaries, MSVC through CMake). Other systems are untested.
Other tokenizers (WordPiece, Unigram), RoPE scaling of the dynamic and longrope kinds, Phi, mixture-of-experts models (including Qwen3.5 35B-A3B), image input, K-quant formats, GPUs and AMX. The full list is under Status in the README.
Generated from docs/faq.md; edit it there.