A small, hand-written Qwen3 inference engine for local hardware — C11 core, no frameworks in the hot path, and every GPU kernel gated against a scalar CPU oracle on real hardware before it counts as working.

Most engines try to run everything. Kipp deliberately runs one pinned model family and aims to run it well enough that you can read the whole engine in an afternoon.

How fast

Qwen3-4B on an Apple M5 Max, against llama.cpp on the same host, the same weights, and the same session — the honest comparison, losing rows included (BF16 prefill re-measured 2026-07-30 alongside Kipp's tensor-path records; decode and quant rows from the 2026-07-28 session, unchanged since):

tokens/s	Kipp	llama.cpp
Decode, BF16	59.7	63.1	llama.cpp +6%
Decode, Q8_0	94.4	100.5	llama.cpp +6%
Decode, 4-bit	127.9 (affine4)	149.6 (Q4_0)	llama.cpp 1.17×*
Prefill, BF16 @2048	3,679	4,102	llama.cpp +12%
Prefill, Q8_0 @2048	1,202	2,729	llama.cpp 2.27×
Prefill, 4-bit @2048	1,210 (affine4)	1,225 (Q4_0)	parity*

* affine4 (scale+bias, gs32) and Q4_0 (scale-only) are the closest available 4-bit schemes, not identical ones — Q4_0 dequantizes more cheaply, affine4 carries a bias term.

The BF16 prefill row is the v0.0.6 story: a gated Metal 4 tensor-ops matmul path (M5-class neural accelerators) took Kipp from 1,312 to 3,679 tok/s — 2.80× in one release — closing what was a 2.9× llama.cpp lead to 12%, with decode untouched and the quantized schemes bit-identical. What remains of that 12% is Kipp's attention stack at prefill, which is the next campaign; quantized projections still run the simdgroup kernels (their tensor variants are future work, hence the Q8_0 prefill row). Every number traces to a committed file in bench/results/, and the measurement protocol explains why they are all same-session medians rather than best-of runs.

Also measured: 175× faster TTFT on a repeated 6,890-token prompt via cross-request prefix reuse, Q8_0 perplexity within 0.02% of BF16, and 4-bit affine weights at 128 tok/s decode.

What it is

• Readable by design — follow one token from embeddings to logits through the linear, model-specific reference pass in src/kipp.c.
• Native all the way down — C11 core, a narrow Objective-C Metal bridge, and isolated CUDA kernels for NVIDIA.
• No frameworks in the hot path — Python and shell are tooling only; PyTorch, JAX, and similar frameworks never enter inference.
• One model family, deeply — closer to whisper.cpp and antirez/ds4 than a generic runner such as llama.cpp or vLLM.
• Correctness first — every GPU backend must match the CPU oracle on real hardware before it is called working. Kernels that no gate exercises are treated as untested, not as working.
• mmap-backed loading — weights are mapped instead of eagerly copied into RAM wherever the backend permits.
• Registry, not runner — every supported checkpoint is pinned to one revision in src/kipp_checkpoints.h; anything else is rejected at load.

What is in v0.0.6

• Backends: scalar CPU oracle, Metal on Apple M5-class hardware, and CUDA validated on H100 for the current default checkpoints (14B/32B on A100).
• Weights: BF16, near-lossless Q8_0, and Q4-class affine 4-bit projections.
• KV cache: paged 32-position blocks, cross-request prefix sharing, and an opt-in Q8_0 KV cache (~1.9× smaller; a memory feature, not a speed one).
• Serving: OpenAI Completions and Chat Completions, SSE, 32-way batched decode, full sampling, Prometheus metrics.
• CLI: greedy or sampled generation, multi-turn chat, perplexity scoring, prompt-lookup and draft-model speculation.

Quickstart

Prerequisites: a C11 compiler, uv for the Python tooling, and Xcode command-line tools for the Metal build. The tooling environment includes PyTorch, because weight conversion reads the original safetensors — inference itself never touches it.

git clone https://github.com/Polished-Snow/Kipp.git
cd Kipp
make quickstart   # downloads + converts Qwen3-0.6B (~1.5 GB), then generates

That is clone to first token in one command. It uses the smallest registered checkpoint so the first run costs a couple of gigabytes rather than the ~16 GB the 4B default needs, and picks Metal on macOS or the CPU path elsewhere.

Or install a prebuilt binary (macOS arm64, Linux x86_64)

Releases ship tarballs with the CLI and server binaries, so inference needs no toolchain at all — the Python tooling is only required to convert weights:

curl -fsSLO https://github.com/Polished-Snow/Kipp/releases/latest/download/kipp-macos-arm64.tar.gz
curl -fsSLO https://github.com/Polished-Snow/Kipp/releases/latest/download/SHA256SUMS
shasum -a 256 --check --ignore-missing SHA256SUMS
tar -xzf kipp-macos-arm64.tar.gz   # kipp, kipp-metal, kipp-server, kipp-server-metal

On Linux, substitute kipp-linux-x86_64.tar.gz (CPU binaries; build the CUDA backend from source with make cuda-generic). The macOS binaries are ad-hoc signed and need macOS 14 or newer. The curl path above just works; if you download through a browser instead, clear Gatekeeper's quarantine flag first with xattr -dr com.apple.quarantine kipp-macos-arm64.

Build the individual binaries directly if you prefer:

make cpu          # session-backed CPU CLI
make metal        # Apple Metal CLI
make server-metal # completions/chat/metrics server on Metal
make test         # hermetic unit tests; no model download (needs uv)

Converting a larger checkpoint is explicit, because it downloads and rewrites multiple gigabytes. CHECKPOINT selects any id from the registry:

make CHECKPOINT=qwen3-4b-base convert   # ~8 GB download, ~7.5 GiB GGUF

Then run a decode — greedy by default, or sampled with a seed:

build/kipp-metal --backend metal \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf \
  --prompt "Hi 世界" --decode 8

build/kipp-metal --backend metal \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf \
  --prompt "Once upon a time" --decode 64 \
  --temperature 0.8 --top-p 0.95 --seed 7

The real-model correctness gates need a converted checkpoint and its pinned vectors:

make test-cpu-model                          # complete CPU model suite
make test-metal                              # complete Metal suite
make test-server
make CHECKPOINT=qwen3-0.6b-base test-model   # any registry checkpoint

Or serve the OpenAI Completions subset on loopback:

build/kipp-server-metal --backend metal --port 8080 \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf

curl -s http://127.0.0.1:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}'

The Metal binary embeds standalone MSL and compiles it through Metal's runtime API. Backend selection is explicit and never falls back to CPU.

More runnable CLI, chat, streaming, and metrics requests live in examples/.

Repository tour

src/kipp.c              strict GGUF loading, tokenizer, sampling, CPU oracle
src/kipp.h              public model, tokenizer, sampling, KV-session C API
src/kipp_backend.h      fixed internal backend boundary + runtime config
src/kipp_checkpoints.h  the supported-checkpoint registry (the growth point)
src/kipp_chat.c         native Qwen3 ChatML rendering
src/kipp_spec.c         prompt-lookup speculative drafting
src/kipp_kv_pool.c      production KV block allocation and prefix sharing
src/kipp_cli.c          greedy/sampled decode CLI with timing metrics
src/kipp_server.c       completions/chat server: batching, SSE, metrics
src/metal/kipp_metal.m  the only Objective-C in the project (Metal bridge)
metal/kipp_kernels.metal standalone, build-time-embedded MSL kernels
src/cuda/kipp_cuda.cu   CUDA bridge, isolated from every other backend
cuda/kipp_kernels.cu    CUDA kernels validated on NVIDIA A100/H100
tests/                  native + server tests, generated reference vectors
tools/                  Python/shell tooling; checkpoints.py mirrors the registry
examples/               copy-pasteable CLI and HTTP requests
docs/                   design docs + the documentation site
AGENT.md                contribution constraints, for humans and AI agents alike

Roadmap

Phase	Goal
0	Pin and specify Qwen3-4B-Base
1	CPU reference path: loader, tokenizer, forward pass, test vectors
2	Correct incremental decoding with a contiguous KV cache
3	Metal backend, validated against CPU reference outputs
4	Isolated CUDA backend, validated on NVIDIA hardware
5	Continuous batching, scheduler, and cross-session KV sharing
6	OpenAI Completions and Chat Completions subset
7	Reviewed extensions: quantization and speculative decoding

Full details are in the roadmap. Deferred work includes ROCm/HIP, extra model families, and arbitrary-GGUF compatibility.

Documentation

The full documentation lives at polished-snow.github.io/Kipp:

• Architecture — design constraints, backends, and correctness boundaries
• Roadmap — correctness-gated implementation phases
• Model support — what supporting a checkpoint means
• Benchmarks — measurement policy and reference results
• Reproducing — exact build, gate, and benchmark commands
• Releasing — release checklist and hardware policy
• Research — reference repositories and design notes
• Contributing — ground rules for human and AI contributors

Contributing

Read AGENT.md first. It is the source of truth for every contributor: keep a C11 core, isolate backends, preserve mmap-backed loading, test on real hardware before claiming success, and never vendor code from the reference repositories.

Security

Kipp's server is a hand-written C11 HTTP server with its own HTTP and JSON parsers. It binds loopback only and has no authentication or TLS by design — see SECURITY.md for the threat model and for how to report a vulnerability privately.

Citing

If you use Kipp or its measurements in academic work, see CITATION.cff (GitHub renders a "Cite this repository" button from it).

Enjoying Kipp?

Kipp is built in the open by Polished Snow. If you find it useful, star the repository and follow project updates:

 

License

Kipp is open source under the MIT License.

_{Built by Polished Snow · X · LinkedIn}