Skip to content

Home

Jump to bottom
github-actions[bot] edited this page Nov 5, 2026 · 39 revisions

Krafla — Portable INT8/INT4 LLM Inference for Legacy x86-64 and ARM CPUs (C11)

CI License: MIT C11 Docs DOI

Krafla (Kernels Retargeted Across Families of Legacy Architectures; formerly isa-lm) is a dependency-free C11 inference runtime for Llama-style decoder Transformers with INT8 and INT4 weights. It runs on x86-64 CPUs from SSE2 up to AVX-512 VNNI, and on ARM CPUs from ARMv7 NEON up to i8mm and SVE.

  • Decoding at least as fast as llama.cpp at every ISA level measured, on the same model and formats: 1.2–1.5× with AVX-512 VNNI, 1.0–2.3× with AVX2, 2.1–10× with SSE2 and SSE4.1, 1.04–1.7× on AArch64 (comparison).
  • Old and low-power CPUs: builds for any x86-64 CPU (SSE2) and for 32-bit ARMv7 NEON boards, with no GPU or external libraries required; static binaries for each ISA.
  • Q4 quality: each Q4_0 block's scale is fitted rather than rounded from the block's maximum. Against llama.cpp's Q4_0 on the same tensors, the KL divergence from the unquantized model is 6–9% lower on four Qwen models and 5% higher on Llama 3.2 1B (Q4_0 block scales). Q8 matches an F32 reference.
  • Deterministic: output is bit-identical for any thread count and prompt batch size.
  • Models: Llama 2/3, Mistral, Qwen2.5, Qwen3, Qwen3.5/3.6, SmolLM2, SmolLM3, Gemma 2 and Gemma 3 from Hugging Face safetensors, with their tokenizers and chat templates.
  • OpenAI-compatible server: ml_server serves a model to the openai client libraries and other tools that take a base URL.

The code is written against a portable baseline. You choose the instruction set (ISA) when you compile, and each operator is bound to the best kernel for that ISA, with no runtime dispatch:

make ISA=avx2                                   # build/avx2/{ml_run,ml_server,ml_bench,test_*}
make ISA=neon CROSS=aarch64-linux-gnu-          # cross build; tests run under qemu
make ISA=avx512vnni THREADS=pthread             # with the worker pool
ISA Minimum CPU Accelerated
scalar any reference implementation of all ops
sse2 any x86-64 (Athlon 64, Core 2) all ops, packed int8 matvec (pmaddwd)
sse41 Penryn / Nehalem int8 matvec (pmaddubsw)
avx2 Haswell / Zen all ops, FMA; f16 packed scales (F16C)
avxvnni Alder Lake / Zen 5 int8 matvec (vpdpbusd ymm)
avx512vnni Cascade Lake / Ice Lake / Zen 4 matvec, quantize, norms, dot, axpy
neon ARMv7 + NEON, any AArch64 (Raspberry Pi 2–4) all ops
neon_dotprod Cortex-A55/A76 (Raspberry Pi 5), Apple M1 int8 matvec (sdot)
neon_i8mm Cortex-A710, Neoverse N2/V1, Apple M2 prompt prefill (smmla)
sve Neoverse V1/N2, Graviton 3 matvec, quantize, norms, dot, axpy (any vector length)

If you run a binary on a CPU that lacks its ISA, it exits with code 3 and names the missing feature. It does not crash with SIGILL.

Quick start

Prebuilt binaries for Linux (x86-64, AArch64, ARMv7) and Windows (x86-64) are on the releases page; their ml_run wrapper starts the fastest build your CPU supports. Fifteen converted models, from SmolLM2 135M to Qwen3.5 9B, are in the Krafla collection on Hugging Face (list), so a first run needs no Python:

# in the unpacked release
curl -LO https://huggingface.co/jonpark0/Qwen2.5-0.5B-Instruct-krafla/resolve/main/Qwen2.5-0.5B-Instruct-Q4_0.mlb
curl -LO https://huggingface.co/jonpark0/Qwen2.5-0.5B-Instruct-krafla/resolve/main/tokenizer.mlt
./ml_run Qwen2.5-0.5B-Instruct-Q4_0.mlb -z tokenizer.mlt --chat qwen -t 4

To build from source:

pip install numpy
make ISA=avx2

# llama2.c checkpoint + tokenizer
python3 tools/convert.py llama2c stories15M.bin stories15M.mlb --wtype q8
build/avx2/ml_run stories15M.mlb -z tokenizer.bin -i "Once upon a time" -n 128 -T 0.8

# Hugging Face Llama, Mistral, Qwen2, Qwen3, Gemma 2 or Gemma 3 (safetensors), and its tokenizer.json
python3 tools/convert.py hf ./Llama-3.2-1B-Instruct llama.mlb --wtype q4 --q8 v,o,down
python3 tools/convert_tokenizer.py ./Llama-3.2-1B-Instruct/tokenizer.json llama.mlt
build/avx2/ml_run llama.mlb -z llama.mlt --chat llama3 -t 4     # one message per line
build/avx2/ml_run qwen.mlb -z qwen.mlt --chat qwen -t 4         # Qwen2.5 Instruct

# OpenAI-compatible API on http://127.0.0.1:8080/v1 (docs/server.md)
build/avx2/ml_server qwen.mlb -z qwen.mlt --chat qwen -t 4

FAQ

Can I run an LLM on a CPU without AVX?

Yes. The sse2 build runs on any x86-64 CPU and the sse41 build on Penryn, Nehalem and later, both with hand-written int8 SIMD kernels.

Is it faster than llama.cpp?

Decoding is at least as fast at every ISA level measured, and faster at most (above). llama.cpp does much more: GPU back ends, K-quants and more architectures. See the comparison.

Does it run on a Raspberry Pi?

The neon builds match the Raspberry Pi 2 to 4 and neon_dotprod the Raspberry Pi 5. They are tested under qemu and on GitHub's Arm runners, not yet on a Raspberry Pi.

How much memory does a model need?

About the size of its .mlb file plus the KV cache: in Q4, Qwen2.5 0.5B takes 265 MiB and Llama 3.2 1B 663 MiB.

More answers in the FAQ.

Documentation

The documentation site and the wiki are both generated from docs/.

Document Contents
FAQ CPUs without AVX, Raspberry Pi, speed against llama.cpp, models, memory, limits
User guide building, cross builds, converting models, CLI options, threads, limits
Server ml_server: the OpenAI-compatible HTTP API, request fields, security
Architecture layers, compile-time kernel binding, ISA guard, data flow, memory, threading, numerics
File format the .mlb model format (header, tensor layout, quantization rule, RoPE scaling) and the .mlt tokenizer format
Writing kernels adding a kernel or an ISA, kernel rules, per-ISA idioms, assembly overrides
Testing test layers, qemu, sanitizers, CI, benchmarks
Comparison with llama.cpp speed per ISA level (x86-64 and AArch64) and perplexity against llama.cpp on the same model and formats
Design spec original design and implementation plan

Status

Area Status
x86 builds all six ISAs tested natively on an AVX-512 VNNI machine: kernel tests against scalar and double-precision references, end-to-end logits against a numpy reference. CI tests scalar through avx2 natively, and builds avxvnni/avx512vnni (SKIP when the runner CPU lacks them)
ARM builds tested under qemu (ARMv7 NEON, AArch64 NEON, dotprod, i8mm, SVE at 128-bit and 512-bit vector length) and on native ARM hardware in CI (GitHub's Arm runners ubuntu-24.04-arm: scalar, neon, neon with threads, dotprod, i8mm, sve)
Real model CI downloads llama2.c stories15M, converts it to Q8, generates text, and checks scalar and avx2 logits (packed and row-major) against the numpy reference: same argmax, top-5 agreement, KL ≤ 1e-2
ISA guard rejects unsupported CPUs with exit code 3, checked under qemu on core2duo/Nehalem/SandyBridge/Haswell and cortex-a53/a76/neoverse-n2
Performance weights repacked into row-interleaved layouts for every SIMD build from SSE2 and ARMv7 NEON up (Q4 decoding up to 2.2× faster); prompts prefilled in batches of up to 128 tokens (2–6× faster), on smmla with i8mm; software prefetch of the weight stream on x86 (decoding 1.3–2.5× faster on one core); attention 2.4× faster on long contexts, 3.4× with the optional Q8 KV cache (--kv q8); an F16 KV cache (--kv f16) halves its memory at the same speed; numbers in Testing
Thread pool bit-identical results for any thread count; ThreadSanitizer clean; a 4-layer dim-2048 Q4 model decodes at 140 tok/s with -t 1 and 503 tok/s with -t 4 (avx512vnni, 4 vCPUs)
Server ml_server answers the OpenAI API (/v1/chat/completions, /v1/completions, /v1/models, streaming) one request at a time, keeping the KV cache between requests; tested over HTTP in every e2e run, including against the openai client
Windows MSVC builds (scalar, SSE2, AVX2) pass the unit tests and match the numpy reference in CI; the MinGW-built release binaries are checked under Wine before publishing
Quantization Q8_0, Q4_0, and Q4 with chosen matrix classes in Q8 (--q8 v,o,down: Llama 3.2 1B perplexity 17.89 at 831 MiB, against 17.27 for Q8 at 1252 MiB and 19.91 for Q4 at 663 MiB); f16 block scales by default (Q4 at 4.5 bits per weight); each Q4 block's scale is fitted rather than rounded from its maximum (against llama.cpp's Q4_0, KL divergence 6–9% lower on four Qwen models and 5% higher on Llama 3.2 1B, see Q4_0 block scales), except for Gemma 3, whose checkpoints do better with llama.cpp's rounding (its quantization-aware 1B checkpoint: Q4 perplexity 40.50, below its Q8); tools/perplexity.py measures it
Models Llama 2/3/3.1/3.2, TinyLlama, SmolLM2, SmolLM3 (layers without RoPE), Mistral (with its sliding window), Qwen2/2.5 (q/k/v biases), Qwen3 (its own head size, per-head q/k norm), Qwen3.5 and Qwen3.6 dense (linear-attention layers with the gated delta rule, a gated attention output, RoPE on a quarter of each head; Qwen3.5 0.8B in Q8_0 scores 15.80 against 15.79 for llama.cpp's Q8_0 and decodes 1.28× faster), Gemma 2 (soft-capped attention scores and logits) and Gemma 3 (GELU, block output norms, local and global layers; the text model of the 4B and larger image-text checkpoints); RoPE scaling of the llama3, linear and yarn kinds. TinyLlama 1.1B, Llama 3.2 1B, Qwen2.5 0.5B, Qwen3 0.6B, SmolLM2 135M and Gemma 3 1B and 4B were run end to end from their Hugging Face files; Q8 perplexity matches an F32 reference (Qwen2.5 18.98 against 19.00, Qwen3 27.90 against 28.10, Gemma 3 41.97 against 42.32)
Tokenizers llama2.c tokenizer.bin; Hugging Face tokenizer.json for SentencePiece-style BPE (Llama 2, TinyLlama, Mistral), Llama 3 byte-level BPE, Qwen2/Qwen3 byte-level BPE with NFC, Qwen3.5 byte-level BPE (marks kept with letters), and GPT-2 byte-level BPE (SmolLM2), converted by tools/convert_tokenizer.py and identical to the tokenizers library on every text tested
Not implemented other tokenizers (WordPiece, Unigram), RoPE scaling of the dynamic and longrope kinds, partial rotary embeddings outside Qwen3.5, image input (of Gemma 3's image-text checkpoints only the text model is converted), architectures other than Llama, Mistral, Qwen2, Qwen3, Qwen3.5, SmolLM3, Gemma 2 and Gemma 3 (Phi, mixture of experts)

Citing

If Krafla helps your research, please cite it (GitHub's Cite this repository uses CITATION.cff):

@software{krafla2026,
  author  = {Park, Yeochoon},
  title   = {Krafla: a portable C11 runtime for INT8/INT4 Llama-style language model inference on legacy x86-64 and ARM instruction sets},
  year    = {2026},
  version = {0.9.1},
  doi     = {10.5281/zenodo.23880653},
  url     = {https://github.com/JonPark0/krafla}
}

Every release is archived on Zenodo. The DOI 10.5281/zenodo.23880653 stands for all versions and resolves to the latest one; each release also has its own DOI, listed on the Zenodo page.

Please also cite the models you run; their licenses are separate from this one.

한국어 요약

Krafla(크라플라, 옛 이름 isa-lm)는 INT8/INT4로 양자화한 Llama 계열 언어 모델을 CPU에서 실행하는 C11 추론 런타임입니다. 이름은 지열 발전소가 있는 아이슬란드의 화산 칼데라에서 따왔고, Kernels Retargeted Across Families of Legacy Architectures의 약자이기도 합니다. GPU나 외부 라이브러리 없이 구형·저전력 프로세서까지 지원하는 것이 목표입니다.

  • 지원 명령어 집합: x86-64는 SSE2부터 SSE4.1, AVX2, AVX-VNNI, AVX-512 VNNI까지, ARM은 ARMv7 NEON부터 NEON dotprod, i8mm, SVE까지 지원합니다. 빌드할 때 명령어 집합을 고르면, 연산마다 쓸 커널이 그때 정해집니다. 지원하지 않는 CPU에서 실행하면 비정상 종료하지 않고, 빠진 기능을 알린 뒤 종료 코드 3으로 끝납니다.
  • 양자화: Q8_0, Q4_0, 일부 행렬만 Q8로 두는 혼합 정밀도를 지원합니다. KV 캐시도 F16이나 Q8로 줄일 수 있습니다.
  • 속도: 명령어 집합별 정수 내적에 맞춘 가중치 재배치, 프롬프트 일괄 처리, 소프트웨어 프리페치, GQA 블록 어텐션을 씁니다. 같은 모델과 형식으로 비교하면 측정한 모든 명령어 집합에서 디코딩이 llama.cpp와 같거나 더 빠릅니다(AVX-512 VNNI 1.2–1.5배, AVX2 1.0–2.3배, SSE2·SSE4.1 2.1–10배, AArch64 1.04–1.7배). 스레드 수를 바꿔도 결과는 비트 단위로 같습니다.
  • 구형·저전력 기기: AVX가 없는 x86-64 CPU(Core 2, Athlon 64)와 32비트 ARMv7 보드, 라즈베리 파이 2–5에 맞는 빌드가 있습니다(실제 라즈베리 파이에서는 아직 측정하지 않았습니다).
  • 모델과 토크나이저: llama2.c 체크포인트와 Hugging Face safetensors(Llama 2, Llama 3/3.1/3.2, TinyLlama, SmolLM2, SmolLM3, Mistral, Qwen2/Qwen2.5, Qwen3, Qwen3.5/3.6, Gemma 2, Gemma 3)를 변환합니다. Hugging Face tokenizer.json도 변환하며, 인코딩 결과는 tokenizers 라이브러리와 같습니다.
  • 사용: 텍스트 생성, 대화 모드(--chat), top-k·top-p 샘플링과 반복 페널티, 퍼플렉서티 측정을 지원합니다. ml_server는 OpenAI 호환 API(스트리밍 포함)로 모델을 제공합니다. 변환된 모델 15개(SmolLM2 135M부터 Qwen3.5 9B까지)는 Hugging Face의 Krafla 컬렉션에서 바로 받을 수 있습니다.
  • 검증: 커널 출력은 스칼라 구현, 배정밀도 계산과 비교하고, 모델 전체의 출력은 numpy 참조 구현과 비교합니다. ARM과 구형 x86은 qemu에서, 실제 ARM 하드웨어는 CI(GitHub Arm 러너)에서 확인합니다.

자세한 문서는 문서 사이트와 위키를 참고하세요.

License

MIT. Third-party notices (llama2.c, Cephes, Unicode data) are in THIRD_PARTY_NOTICES.md.

Generated from README.md; edit it there.

Krafla

Clone this wiki locally