A CPU-based LLM inference engine built from scratch in C++.
The goal of this project to implement the full answer generation operation of an llm model without relying on premade machine learning software that are normally used by machine learning engineers as blueprints to build such llms. These "blueprinting software" packs are often called frameworks and one such popular framework is PyTorch, made to be used in python.
Instead of using PyTorch we will be implementing every operation ourselves, then optimizing for speed through hardware specific tricks.
For now, we will be building an inference engine for the GPT2 model, meaning, we will build it based on the publically available weights resulting from the GPT2 training process of OpenAI.
GPT stands for Generative Pretrained Transformer.
- Genrative because it generates new text.
- Pretrained because it's been pretrained on massive amounts of text and learned how words an concepts fit together.
- Transformer because the model transforms input data (like a prompt) into mathematical representations, and then back into output data (like a response)
Compared to current LLMs it's a pretty bad model at most tasks you'd typically use an AI chatbot for, but it's architecture is well documented, simple and essentially very similar to more powerful models.
We're building in such a way so that when it's ready, with some small changes, we can run models like Meta's Llama, which is a proper llm model that can answer knowledge questions and even write code.
This is both a learning project and a stepping stone toward a local, private, fully self-hosted LLM product.
When you use ChatGPT or Claude, two things happen:
- Training — the model learns from billions of words (done once, costs millions)
- Inference — the model generates a response to your prompt (happens every time)
An inference engine is the software that runs inference, it generates a response to a prompt.
It's a system, think of it as a black box that takes your prompt as input and produces the answer as output. If you zoom into that black box you'll see a series of smaller black boxes that take the input and transorm it, meaning, change it in some mathematical way using matrix multiplication, using GPT2 weights as parameters to do the correct transformations, and pass it on to the next black box, and so on till we get the desired output. We'll call these black boxes blocks and in code they'll usually be implemented using the programming concept of a Class.
PyTorch can do inference, but it's built for flexibility and research, not for squeezing maximum performance out of specific hardware. We're building a leaner, faster alternative that runs entirely on CPU (not GPU!) with no ML framework dependency when being run.
flowchart LR
A[Prompt] --> B[Tokenizer]
B --> C[Model Block]
C --> D["Logits\n(probability of next token)"]
D --> E[Choose Most likely Token]
E -->|add token| C
E -->| Genreation done | F[Detokenizer]
F --> G[Answer]
classDef done fill:#22c55e,stroke:#16a34a,color:#fff
classDef todo fill:#f1f5f9,stroke:#94a3b8,color:#334155
class A,B,F,G done
class C,D,E todo
-
Tokenizer — splits your prompt into tokens (subword chunks) and converts them to integer IDs. For example: "Hello world" → [15496, 995]
-
Model Block — runs the forward pass, the series of matrix multiplications. Takes token IDs, produces logits (a probabillity score for every possible next token)
-
Choose Token — pick the highest scoring token
-
Repeat — add the new token back to the input and run again until the model decides a response has been fully generated.
-
Detokenizer — convert token IDs back to text
The following section explains the architecture at a high level. The mathematical details will be introduced only when implementing each component.
GPT-2 is a Transformer-based language model. Internally, it is not one single operation, but a sequence of repeated Transformer blocks.
The GPT-2 model we are implementing is GPT-2 Small, which contains:
- 12 Transformer blocks
- 12 attention heads per block
- 768-dimensional internal representations
- ~124 million learned parameters (weights)
The weights are the result of the original GPT-2 training process. Our goal is not to train GPT-2 again, but to recreate the process of using those weights to generate text.
At a high level, GPT-2 works like this:
flowchart TD
A[Input Text]
A -->|split text into smaller pieces| B[Tokenizer]
B -->|convert pieces into numbers| C[Token IDs]
C -->|prepare input for the neural network| D[Transformer Blocks × 12]
D -->|calculate scores for every possible next token| E[Token Scores]
E -->|select the next token| F[Generated Token]
F -->|append token and continue generation| D
The generated token is added back to the input sequence and the process repeats. This is called autoregressive generation.
The Transformer blocks are where the majority of the model computation and "intelligence" happens. Each block performs the same sequence of operations:
flowchart TD
A[Input Information]
A --> B[Layer Normalization]
B -->|find relationships between tokens| C[Self Attention]
C --> D[Add]
A -.->|residual connection| D
D --> E[Layer Normalization]
E -->|process and transform information| F[MLP<br/>Feed Forward Network]
F --> G[Add]
D -.->|residual connection| G
G --> H[Output Information]
Every Transformer block receives information about the text, transforms it, and passes an improved version to the next block. GPT-2 becomes powerful because it repeats this same process many times while using different learned weights in each block.
1. Self Attention
Attention is the mechanism that allows the model to decide which previous tokens are important when processing the current token.
For example:
"The animal didn't cross the street because it was tired."
To understand the meaning of "it", the model needs to connect it to "animal". Attention allows the model to learn relationships between tokens and use this information when generating text.
GPT-2 uses causal attention, meaning each token can only look at previous tokens. It cannot look into the future.
This restriction is what allows GPT-2 to generate text one token at a time.
The MLP is a small neural network inside each Transformer block.
While attention focuses on relationships between tokens, the MLP performs additional transformations on the information extracted by attention.
A simple way to think about it:
- Attention decides what information is important.
- The MLP processes and transforms that information.
Residual connections allow information to bypass parts of the network.
Instead of forcing every layer to completely replace the input, the model adds the original information back after each transformation.
This helps preserve useful information and makes very deep networks easier to train.
4. Layer Normalization
Layer normalization keeps the numerical values flowing through the network stable.
Without normalization, values could become too large or too small, making the model difficult to train and use.
The important idea is that every Transformer block is almost identical. GPT-2 becomes powerful by repeating this block many times and learning the correct weights during training.
Our inference engine will recreate this process:
- Load GPT-2's pretrained weights.
- Convert input tokens into numerical representations.
- Run the input through all Transformer blocks.
- Produce scores for possible next tokens.
- Select the next token and repeat.
flowchart TD
A[frontend.py Python\nTokenizer + UI] -->|token ids| B[engine C++\nInference Loop]
B -->|token ids| C[Model Interface]
C -->|implemented by| D[MockModel\ntemporary]
C -->|will be replaced by| E[GPT2Model\nreal C++ inference]
E --> F[Tensor\nweight storage + math]
F --> G[Weights\nbinary files on disk]
frontend.py — Python script. Handles user input, tokenizes prompts using a ready made tokenizer, sends token IDs to the engine , decodes the output back to text.
engine (C++) — the main program. Reads token IDs from input, passes them to the Model interface, writes output token IDs back to the python input. Owns the autoregression loop.
Model (abstract class) — the interface every model implementation must
satisfy. It acts like a black box,performing a function but hiding how things actually work from behind so we can change how the model works easily (for example later swapping GP2 for Llama will be way easier becaue of this Model abstract class). The interface only "exposes" (ie doesnt hide) one capability: A function called generate(token_ids) → token_ids.
A function is basically a subprogram and the function generate takes the tokenized prompt, transforms it, and output the tokenized response.
The engine never knows which model it's realy running.
MockModel — temporary implementation of Model. Runs a Python script/program (mock_model.py) that runs real GPT-2 using PyTorch. Lets us test
the full system end to end before the real C++ inference is ready.
GPT2Model — the real implementation. Will replace MockModel with no changes to the engine or frontend. Implements the full GPT2 transformer forward pass (the inference) in C++ using the operations we will build on our own.
Tensor — our core data structure. A 3D array of weights accompanied by all the math operations the forward pass needs (matrix multiplication,matrix addition, the softmax operation etc.)
Weights — stored in binary files dumped from PyTorches pretrained GPT-2 using
scripts/analyze.py. Loaded once at startup into Tensor objects.
| Phase | What | Status |
|---|---|---|
| 1 — Engine Setup | Tensor class, weight loading, pipeline skeleton | Done |
| 2 — Forward Pass | Embedding lookup, layernorm, autoregression loop | In progress |
| 3 — Attention Block | Multi-head attention, QKV, causal mask, softmax | Todo |
| 4 — MLP Block | Linear layers, GELU | Todo |
| 5 — Optimization | Tiled matmul, threading, KV-cache, quantization | Todo |
First, clone the repository:
git clone <repository-url>
cd inference-engineCreate a Python environment and install the required dependencies:
python3 -m venv .venv
source .venv/bin/activate
pip install torch transformersThe first time the project is run, generate the GPT-2 weight files and reference outputs:
python3 scripts/analyze.pyThis extracts the pretrained GPT-2 weights and creates files that the C++ engine will load.
Build and run the project:
make runType a prompt and hit enter.
To exit:
\END
The project uses Make to compile the C++ code.
Make is a tool that runs predefined build commands. Instead of manually typing
compiler commands, we can use simple commands like make and make run.
You need:
- Linux (recommended) or WSL2 on Windows
- C++ compiler (
g++) - Make
- Python 3
On Ubuntu, these can be installed with:
sudo apt update
sudo apt install build-essential python3 python3-venvRunning:
makecompiles the C++ source files into executable programs.
The result is:
bin/
├── engine # main C++ inference engine
└── test # test executable
The engine program contains the inference loop. The Python scripts handle
tokenization and communication with the C++ engine.
Running:
make runautomatically builds the project and starts the Python frontend.
The flow is:
flowchart LR
A[User Prompt] --> B[tokenizer.py]
B -->|token IDs| C[bin/engine]
C -->|generated token IDs| B
B --> D[Text Output]
The Python frontend:
- receives your prompt
- converts it into token IDs
- sends them to the C++ engine
- converts the generated token IDs back into text
The C++ engine:
- runs the inference loop
- executes the model operations
- produces the next tokens
Build the project:
makeRun the project:
make runRun tests:
./bin/testRemove compiled files:
make cleanIf you modify C++ source files, run make again before testing your changes.
- scripts/analyze.py — creates GPT-2 weight files, activations, reference outputs
- scripts/mock_model.py — temporary PyTorch GPT-2 backend
- script/tokenizer.py — Python frontend (prompt loop, tokenize, decode)
- src/engine.cpp — main entry point, autoregression loop
- src/Tensor.hpp/cpp — core tensor class
- src/model/Model.hpp — model interface
- src/model/MockModel.hpp/cpp — temporary PyTorch-backed implementation
This section describes the workflow for making changes to the project.
The goal is to work on the project like a real software engineering team: choose a task, understand the existing code, implement a change, test it, and submit it for review.
Tasks are tracked using issues.
An issue describes one specific piece of work, such as:
- Implement matrix multiplication for Tensor
- Add a weight loading function
- Implement Layer Normalization
- Improve performance of an operation
- Add tests for a component
Before starting:
- Pick an open issue.
- Read the description carefully.
- Ask questions if anything is unclear.
- Comment on the issue so others know you are working on it.
Before writing code:
- Find the files related to the task.
- Read the existing code.
- Understand how the component fits into the rest of the system.
- Decide what needs to change.
Claude can help with understanding the code.
The goal is to understand the change before implementing it.
3. Create a Branch
Each task should be developed on its own branch.
Create a branch:
git checkout -b feature/task-nameExample:
git checkout -b feature/tensor-multiplyA branch is a separate version of the project where changes can be made without affecting the main code.
Make the required changes to the code.
During implementation:
- Keep changes focused on the task.
- Follow the existing code style.
- Use Claude to explain unfamiliar concepts, suggest solutions, and help debug problems.
- Ask for help if the correct design is unclear.
Claude is especially useful for:
- Explaining existing code
- Suggesting implementation approaches
- Finding bugs
- Explaining compiler errors
- Reviewing changes
Try to understand what the generated code does before adding it to the project.
Once the change is complete:
Add your changes:
git add .Create a commit:
git commit -m "Describe what changed"A commit is a saved checkpoint of your work.
Upload your branch:
git push -u origin feature/task-nameA Pull Request (PR) asks for your changes to be reviewed by me and merged into the main project.
The PR should include:
- What was changed
- Why it was changed
- Any design decisions or questions
After review, the changes can be merged into the main project.
A good workflow is:
- Ask Claude to explain the problem.
- Ask for a possible implementation approach.
- Implement the change.
- Ask Claude to review the code.
- Test the result.
Avoid asking Claude to generate large parts of the project without understanding them first. Smaller, understandable changes are easier to debug and improve.
This glossary explains programming, machine learning, and project-specific terms used throughout the documentation.
You do not need to memorize these terms before contributing. Use this section as a reference whenever you encounter something unfamiliar.
A set of instructions that a computer executes to perform a task.
The human-readable instructions written by programmers.
In this project, the source code is mainly written in C++ and Python.
A program that converts source code into a program that the computer can run.
Example:
C++ source code → Compiler → Executable program
A compiled program that can be directly run by the operating system.
Example:
./bin/engine
runs the compiled C++ inference engine.
A reusable piece of code that performs a specific operation.
Example:
generate(token_ids)takes token IDs as input and returns generated token IDs.
A blueprint for creating objects.
A class groups together data and functions that belong together.
A specific instance of a class.
Example:
A Tensor class defines what a tensor is. A Tensor object stores actual
numerical data.
A definition of what a component can do without specifying how it does it.
In this project, the Model interface defines what every model must provide,
without knowing whether the model is GPT-2, Llama, or another model.
The actual code that performs the behavior defined by an interface.
A collection of reusable code written by other developers.
Examples:
- PyTorch
- Transformers
External software that a project requires in order to work.
A tool used to track changes to code and collaborate with other developers.
A folder containing a project's source code and Git history.
A separate version of a repository used to develop changes without affecting the main code.
A saved checkpoint of changes.
A commit records what changed and when.
A request to merge changes from one branch into the main project.
The process of converting source code into an executable program.
Example:
C++ files → Compiler → Executable
The process of finding and fixing problems in a program.
A mathematical system that transforms input data into output data.
In this project, GPT-2 is the model.
The process of adjusting a model's internal values so it learns patterns from data.
Training is usually very expensive and is done before the model is released.
Using a trained model to generate outputs.
This project focuses on inference, not training.
A learned numerical value inside a neural network.
During training, the model adjusts its weights. During inference, these weights are loaded and used to generate outputs.
Another name for a value learned during training.
A model's size is often measured by its number of parameters.
Example:
GPT-2 Small has approximately 124 million parameters.
A data structure used to store multi-dimensional numerical data.
Neural networks perform their calculations using tensors.
Examples:
A list of numbers representing information.
Example:
[0.2, 0.8, -0.4]
A grid of numbers.
Matrix multiplication is one of the main operations used in neural networks.
A small piece of text that an LLM processes.
A token can be a complete word, part of a word, punctuation, or a symbol.
Example:
"playing" → ["play", "ing"]
A program that converts text into tokens and token IDs.
Example:
"Hello" → [15496]
A numerical representation of a token.
The model cannot directly process text, so tokens are converted into numbers that contain useful information.
A processing stage inside a neural network.
A model is usually made by stacking many layers together.
The neural network architecture used by GPT-style models.
A Transformer is made of repeated blocks containing attention and neural network layers.
A mechanism that allows the model to decide which previous tokens are important when processing the current token.
The raw scores produced by the model for every possible next token.
Higher logits mean a token is considered more likely.
A value representing how likely something is to happen.
The model converts logits into probabilities to choose the next token.
Running input data through the model once to produce an output.
In this project, a forward pass takes input tokens and produces scores for the next token.
Generating text one token at a time.
The newly generated token becomes part of the input for the next generation step.
A repeated component inside GPT-2.
Each block contains:
- Self attention
- MLP
- Layer normalization
- Residual connections
GPT-2 gains its power by repeating this block many times.
The C++ implementation that stores numerical data and provides mathematical operations required by the model.
A working implementation used for comparison.
In this project, the PyTorch GPT-2 implementation is the reference. The C++ implementation should produce equivalent results.
Reducing the precision of model weights to make the model smaller and faster.
A technique that stores previous attention calculations during generation so they do not need to be calculated again.
This section explains C++ concepts that appear throughout the project.
You do not need to know these before starting. Use this section as a reference when encountering unfamiliar code.
A file containing declarations that describe what code is available.
Header files usually end with .hpp or .h.
Example:
#include "Tensor.hpp"
This allows the current file to use code defined in Tensor.hpp.
A file containing the actual implementation of code.
Source files usually end with .cpp.
Example:
Tensor.hpp → describes the Tensor class
Tensor.cpp → contains the Tensor class implementation
A statement that allows one file to use code from another file.
Example:
#include <iostream>
#include "Tensor.hpp"
A named location in memory that stores a value.
Example:
int size = 10;
A description of what kind of data a variable stores.
Examples:
int // integer number
float // decimal number
bool // true or false
Tensor // custom type created by us
A blueprint for creating objects.
A class defines:
- what data an object stores
- what operations can be performed on it
Example:
class Tensor {
float* data;
void multiply();
};
A specific instance of a class.
Example:
Tensor input;
Tensor is the class. input is an object created from that class.
A special function that runs when an object is created.
It is usually used to initialize the object's data.
Example:
Tensor input(3, 768);
A function that belongs to a class.
Example:
tensor.matmul(other);
matmul is a method of the Tensor class.
A variable that stores the memory address of another variable.
Example:
float* data;
Pointers are commonly used when working directly with memory.
Another way of referring to existing data without copying it.
Example:
Tensor& input
The function receives the original Tensor instead of creating a copy.
References are commonly used for performance.
A keyword that means something cannot be modified.
Example:
const Tensor& input
The function can read input, but cannot change it.
The process of converting C++ source code into a program that can run.
Example:
.cpp files → compiler → executable
A program that converts source code into an executable.
Example:
C++ source code → g++ → executable program
The process of compiling all source files into an executable.
In this project:
make
builds the C++ inference engine.
A file containing instructions for the Make build system.
It describes:
- which files need to be compiled
- how to compile them
- which commands can be run
A problem found while compiling the program.
Examples:
- missing include
- invalid syntax
- wrong data type
The program cannot be built until compiler errors are fixed.
A problem that happens while the program is running.
Examples:
- invalid memory access
- incorrect calculations
- unexpected behavior
A runtime error caused by accessing memory that the program is not allowed to access.
Common causes:
- invalid pointers
- accessing released memory
- accessing outside an array's size
The process of finding and fixing problems in code.
Common debugging techniques:
- reading error messages
- printing values
- using a debugger
- testing small parts separately
A programming style where code is organized around objects that contain both data and functions.
This project uses OOP for components like:
ModelTensorMockModelGPT2Model
A function exists independently:
calculate();
A method belongs to a class:
tensor.calculate();