Kindi 0.2.0

Kindi 0.2.0 runs the Ternary-Bonsai-2 models, from 2B to 27B, on more hardware: the Quadro K420 and K1200, two
K1200 cards in tensor parallel, the GeForce GTX 1080, the GeForce RTX 3080 and the CPU. The default output stays
token-for-token identical to llama.cpp for the same model file.

Install with one command

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

The installer finds your cards and selects the model and the settings. It downloads the model and checks its
sha256, converts it, checks the output against llama.cpp and starts the server. --dry-run shows each step first.
To check the installer's own sha256, see docs/INSTALL.md.

What is new

• The 27B on one card or on two cards. A new engine runs Ternary-Bonsai-2-27B on a GeForce GTX 1080, on a
  GeForce RTX 3080 or on two Quadro K1200 cards. The K/V cache is q8_0, as llama.cpp with -ctk q8_0 -ctv q8_0.
• The 4B and the 9B. On the Quadro K1200 and on the CPU.
• Prompts in chunks. Kindi reads up to 256 prompt tokens in one pass. The tokens are the same as with one token
  at a time.
• 5-trit weights on Pascal and later. 1.75 bits per weight: the GTX 1080 holds the 27B with a context of 65,536
  tokens.
• Tensor cores and a draft model on the RTX 3080. KINDI_TC=1 uses int8 tensor-core products and CUDA graphs.
  KINDI_SPEC=draft2b uses the 2B as a draft model; the greedy output is the same as without it.
• A CPU engine. KINDI_DEVICE=cpu runs the 2B, 4B and 9B with no CUDA.
• The prompt cache. The memory budget counts all the checkpoints. Saved openings belong to one engine
  configuration. A damaged opening file does not stop a request.
• The version and the source commit on /health.
• docs/CARDS.md gives the settings for each card. docs/BENCHMARKS.md compares Kindi and llama.cpp on the
  same card, with the same model file and the same context.

Speed (tokens/s, exact head, greedy)

Card	Model	Generation (short context)	Prompt reading (512 tokens)
Quadro K420 1 GB	2B	20.8	33.3
Quadro K1200	2B / 4B / 9B	81.0 / 39.2 / 23.4	194.5 / 77.4 / 41.7
2x Quadro K1200	27B	12.6	19.5
GeForce GTX 1080	27B	27.9	254.0
GeForce RTX 3080, 190 W	27B	63.2	1,093
GeForce RTX 3080, 380 W ¹	27B	66.4	1,361

¹ 380 W: dev build release-tcspec6 (2026-10-22), whose exact path is the same code as 0.2.0. The other rows without
a build note are Kindi 0.2.0.

With the same model file and the same tokens, llama.cpp generates 1.65 tokens/s on the K420 (Kindi: 20.8), and it
reads the 512-token prompt at 1.63 tokens/s on the two K1200 cards (Kindi: 19.5). With the 2B draft model, the
RTX 3080 generates about 100 tokens/s on agent prompts at 190 W.

Exactness

On the reference prompts, the greedy output is token-for-token the same as llama.cpp on every card. There is one
exception. On the 27B, the 3,000-token reference differs at token 15. There, the two best tokens are 0.0002 logits
apart on most cards, and at most 0.03 on the RTX 3080's tensor-core path, and the order of the additions
decides.

Upgrade from 0.1.0

Run the installer, or build the image again: docker build -t kindi:0.2.0 -f docker/Dockerfile .. The pack of the
2B from 0.1.0 still works. Saved openings of 0.1.0 are not used again: the engine reads those prompts once more.

Kindi 0.1.0

Kindi 0.1.0 is the first public release. Kindi is a custom inference engine for the NVIDIA Quadro K420, a 2014 workstation card with 1 GB of memory. It runs Ternary-Bonsai-2-2B and has an OpenAI-compatible server.

Headline numbers (one Quadro K420)

	generation	prompt read
llama.cpp (Vulkan build, all layers on the GPU)	2.84 tokens/s	2.99 tokens/s
Kindi 0.1.0, exact head	20.8 tokens/s	26.5 tokens/s
Kindi 0.1.0, fast head	22.9 tokens/s	26.5 tokens/s

• Context: 24,576 tokens in 1 GB.
• With the exact head, the output is token-for-token identical to llama.cpp (kindi verify).

What is in 0.1.0

• kindi convert makes an exact conversion from the PTQ1_0 GGUF.
• An OpenAI-compatible server with chat completions, streams, tool calls, an optional API key and live progress in /health.
• Thinking mode, with the thinking in reasoning_content and a limit through thinking_budget_tokens.
• A prompt cache: checkpoints of the recurrent state, parked conversations in RAM and saved openings on disk.
• An optional fast output head that you can change at run time (kindi head).
• A Docker image with the CUDA 10.2 / CuPy 9.6 stack that Kepler needs.

Requirements

• NVIDIA Quadro K420 (GK107, 1 GB) and the NVIDIA 470 driver.
• Linux x86-64 and Docker with the NVIDIA Container Toolkit, or CUDA 10.2 + Python 3.6 + CuPy 9.6.
• 4 GB RAM. About 1.5 GB of disk for the model and the pack, and about 4 GB for the Docker image.

Known limits

• One request at a time. Other requests wait in a queue.
• Kindi reads new prompt text at about 26 tokens/s. A new 5,000-token prompt takes about 3 minutes. The cache makes repeated prompts fast.
• One model (Ternary-Bonsai-2-2B, PTQ1_0) and one GPU (we tuned and tested Kindi on the K420 only).
• A 2B model makes the mistakes of its size. Check what it writes.

Files

• kindi-0.1.0.tar.gz: the source of this release.
• kindi-demo-untrimmed.mp4: the full demo recording (123 s). No pauses are shortened.
• kindi-demo.cast: the raw asciicast of the same recording.

License

GNU AGPL-3.0. Commercial licenses are available (see COMMERCIAL-LICENSE.md).

Read the README for the install steps. The agent benchmark is kbench. The project page is amirhusain.com/kindi.