Micro-Infer: C++ Neural Runtime

Micro-Infer is a zero-dependency, high-performance Neural Language Model runtime built entirely from scratch in C++20. It demonstrates how to execute a deep learning Multi-Layer Perceptron (MLP) model for character-level text generation without relying on heavy external libraries like PyTorch, Eigen, or BLAS.

🧠 Project Architecture

The project is split into two distinct phases:

1. Python Training Environment (scripts/): We use PyTorch to train a tiny neural network (MLP) on the Shakespeare dataset. Once trained, a custom script (export_weights.py) dumps the raw floating-point weights into a custom binary format (model.bin).
2. C++ Inference Engine (src/, include/): The core of the project. It uses OS-level memory mapping (Win32 CreateFileMapping) to map the model.bin directly into memory instantly. It then runs a custom-built, multi-threaded Matrix Multiplication (GEMM) engine to generate text token-by-token.

---

🚀 How to Run the Project from Scratch

Follow these steps to build, train, and run the inference engine from the very beginning.

Step 1: Build the C++ Engine

You need a C++20 compiler (like MinGW GCC) and CMake installed.

1. Open your terminal in the d:\micro_infer directory.
2. Generate the build files and compile the executable in strict Release mode for maximum optimization:

   # Remove old unoptimized build
   rm -rf build
   
   # Configure CMake specifically for MinGW in Release mode
   cmake -B build -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release
   
   # Build the project using all your CPU cores (e.g., 16)
   cmake --build build -j 16

   This creates standalone executables in the build/bin/ folder.

Step 2: (Optional) Train the Model in Python

We have already provided a pre-trained model in the pretrained/ folder, so you can skip this step if you just want to run the engine. However, if you want to train it yourself from scratch:

1. Ensure you have Python installed with PyTorch.
2. Run the training script (it will automatically download the Shakespeare dataset):

   python scripts/train_mlp.py

3. Export the trained PyTorch weights into our custom C++ binary format:

   python scripts/export_weights.py

Step 3: Run the C++ AI Text Generator

Now run the main executable to generate text!

1. In your terminal, run the compiled binary:

   .\build\bin\micro_infer.exe

2. Customizing the Output: You can change the starting prompt, the length of the text, and the "creativity" (temperature) using command-line arguments:

   .\build\bin\micro_infer.exe --prompt "HAMLET: " --tokens 1000 --temp 0.8

   • --prompt: The starting text to feed the model.*
   • --tokens: How many characters to generate.*
   • --temp: Lower numbers (0.2) make it repetitive and strict, higher numbers (1.2) make it chaotic.*

3. Hitting Maximum Speed (50K+ tok/s): When running interactively in Windows terminals (like VS Code), the C++ engine runs so fast that it gets bottlenecked waiting for the terminal UI to draw the characters on screen. To see the true hardware speed of the engine, redirect the output to a text file to bypass the UI rendering delay:

   .\build\bin\micro_infer.exe > output.txt

   (Open output.txt to read the generated text and see the peak tokens/sec)

Step 4: Run the Test Suite

We built a comprehensive test suite to verify the custom math engine. You can run these to ensure your CPU is computing the matrix math correctly:

# Test 1: Checks zero-copy memory mapping and file parsing
.\build\bin\test_tensor.exe

# Test 2: Checks the math engine (Matrix Multiplication, Softmax, Tanh)
.\build\bin\test_ops.exe

# Test 3: Benchmarks the multi-threaded speed vs a naive loop
.\build\bin\test_bench.exe

---

⚡ Technical Highlights

• Zero-Copy Loading: The model.bin is never copied into RAM using malloc or memcpy. It uses Windows virtual memory mapping, meaning loading is instantaneous and uses 0 extra bytes of RAM.
• Cache-Tiled GEMM: Matrix multiplication is broken down into 32x32 tiles so that the data stays entirely inside the CPU's ultra-fast L1 cache, preventing memory bottlenecks.
• Parallel Multithreading: Uses C++20 std::jthread to split the matrix rows across all available CPU cores (e.g., 16 cores).
• Standalone Binary: The .exe is statically linked. You can copy it to any Windows machine, put it next to model.bin and vocab.json, and it will run instantly without needing any installations.