thinference

thinference is a deliberately minimal SmolLM2-360M-Instruct inference engine, written in scalar C++17 to be read top to bottom and stepped through in a debugger. There is no GPU, no SIMD intrinsics, no threads, and no KV-cache — just the arithmetic of a transformer, laid out so that every number that scrolls past on screen traces back to a single line of source. It decodes greedily (always the single highest-scoring next token), re-runs the whole network from scratch on every step, and is fast enough to sit and watch it think. It is a teaching engine first; a fast engine never.

Read the companion guide

If you know CS and C++ but have never touched machine learning, start with the companion guide: docs/index.html. It explains how LLM inference actually works, concept by concept, from scratch — and uses this exact engine as its worked example, so the prose and the code never drift apart.

Open it in any browser:

open docs/index.html        # macOS
xdg-open docs/index.html    # Linux

…or just double-click the file. The guide is a self-contained offline bundle — docs/index.html together with its stylesheet docs/styles.css, which must stay in the same folder — so it needs no server, no build step, and no internet connection, and works straight off file://. If you would rather read it on paper, there is a Print / Save as PDF button at the top.

Prerequisites

• CMake ≥ 3.25
• Ninja
• A C++17 compiler — a recent Clang, GCC, or AppleClang

One-liners:

# macOS (Homebrew)
brew install cmake ninja

# Debian / Ubuntu
sudo apt install cmake ninja-build build-essential

(If your distribution ships a CMake older than 3.25, install a newer one with pip install cmake or from Kitware's APT repository.)

Python 3 is needed for the model-acquisition workflow below — both Option A (pip-installs the Hugging Face CLI) and Option B (runs llama.cpp's Python converter) use it. Once you have the GGUF, building and running the engine itself needs no Python. pre-commit is needed only if you intend to contribute — see Contributing.

Get the model

The engine loads exactly one file, from a path baked into the source (src/main.cpp):

models/smollm2-360m-instruct-f16.gguf

Keeping the engine minimal means there is no --model flag — the path is fixed. So mind the casing trap: the filename must be exactly that, all lowercase. On a case-sensitive filesystem (e.g. Linux), SmolLM2-360M-Instruct-f16.gguf will not be found — rename it to lowercase so it works everywhere.

Option A — download a ready-made F16 GGUF (quickest)

The official HuggingFaceTB/SmolLM2-360M-Instruct-GGUF repo only ships a q8_0 quant, so we pull the half-precision (F16) build from a community mirror:

python3 -m pip install -U huggingface_hub
hf download bartowski/SmolLM2-360M-Instruct-GGUF \
    SmolLM2-360M-Instruct-f16.gguf --local-dir models
mv models/SmolLM2-360M-Instruct-f16.gguf models/smollm2-360m-instruct-f16.gguf

That repo's SmolLM2-360M-Instruct-f16.gguf was the exact 726 MB F16 file as of June 2026. Community repos can move; if it has vanished, use Option B, which always works.

Option B — convert the official weights yourself (canonical fallback)

This path starts from the original, first-party checkpoint and will keep working regardless of what any mirror does:

# 1. The original Hugging Face checkpoint (safetensors)
python3 -m pip install -U huggingface_hub
hf download HuggingFaceTB/SmolLM2-360M-Instruct \
    --local-dir smollm2-360m-hf

# 2. llama.cpp's converter turns it into an F16 GGUF
git clone https://github.com/ggml-org/llama.cpp
pip install -r llama.cpp/requirements.txt
python3 llama.cpp/convert_hf_to_gguf.py smollm2-360m-hf \
    --outtype f16 --outfile models/smollm2-360m-instruct-f16.gguf

Option B writes straight to the correct lowercase name, so there is nothing to rename.

Confirm it landed

Either way, the result should be about 692 MiB (726 MB):

ls -lh models/smollm2-360m-instruct-f16.gguf

A note on memory: the on-disk file is ~692 MiB, but the engine widens every weight from F16 to F32 as it loads (~1.35 GiB of owned buffers) while keeping the original file mapped for the whole run. Budget roughly 2+ GiB of free RAM — a machine with room for the 692 MiB file on disk may still be short on memory to run the engine.

Quickstart: build & run

cmake --preset release
cmake --build --preset release
./build/release/thinference --prompt "What is the capital of France?"

The run announces the model, echoes the tokenized prompt, then narrates each greedy step — the token it chose and the top few it weighed — before printing the finished reply on its own >-prefixed line. Abridged, it looks like this:

[00m02s] Loaded model: 32 layers, vocab 49152
[00m02s] Encoded chat ids: [1, 4093, 198, 1780, 314, 260, 3575, 282, 4649, 47, 2, 198, 1, 520, 9531, 198]
[00m03s] Choosing "The". Candidates: ["The"=0.932329, ...]
...
[00m20s] Choosing " Paris". Candidates: [" Paris"=0.99868, " the"=0.000224566, " named"=0.000171473]
[00m20s] > The capital of France is Paris.

(The elapsed-time tags and exact timings will differ on your machine. The token ids should match; the last digits of the probabilities may vary under -ffast-math, LTO, or auto-vectorization, but the greedy tokens shown here have been verified on the tested builds.)

Use -n / --max-new-tokens to change the token budget (default 64), and -h for the full usage:

./build/release/thinference --prompt "Write a haiku about pointers." -n 32

Debug vs. release

There are two presets, and the choice matters:

• release — -O3 plus link-time optimization and -ffast-math. This is the one to watch it generate, at roughly 1 token/second. (Verified: the faster vectorized math does not change this model's greedy output.)
• default (Debug) — unoptimized, so a debugger steps through the exact, unreordered arithmetic the source describes, in source order. This is the one to understand the engine with. It is much slower than release — a Debug run is not ~1 tok/s, so when it sits there churning, it has not hung; it is just doing every multiply the honest way.

cmake --preset default
cmake --build --preset default
./build/default/thinference --prompt "What is the capital of France?"

Where to start reading the code

Begin with src/main.cpp — the end-to-end driver reads top to bottom: load the model, tokenize the prompt, generate one token at a time, print. From there, follow the same path the guide takes through the pieces it wires together:

gguf  →  config  →  weights  →  tokenizer  →  model / kernels

That is: parse the GGUF container, read the hyperparameters out of it, map the weight tensors, build the tokenizer, then run the transformer (model.cpp) on top of the scalar math primitives (kernels.cpp).

Contributing

The code explorer embedded in docs/index.html is a generated snapshot of src/. After you change any file under src/, regenerate it and commit the updated guide alongside your code change:

python3 docs/embed_sources.py

Formatting is handled by clang-format via a pre-commit hook. Install it once:

pip install pre-commit
pre-commit install

After that, your commits are auto-formatted to match the project style.

License

The source in this repository is released under the MIT License — see LICENSE.

The SmolLM2-360M-Instruct weights you download are not part of this repository and carry their own upstream license (Apache-2.0, from HuggingFaceTB). Respect their terms when you redistribute them.