mini-tensorrt

A Lightweight C++17 Deep Learning Inference Engine

Overview

mini-tensorrt is a minimal deep learning inference engine built from scratch in modern C++17.

Key Features

• Modern C++17: RAII, smart pointers, STL containers
• Tensor Operations: Efficient float32 tensor with row-major layout
• Layer Support: Dense (fully-connected) and ReLU activation layers
• Flexible Models: JSON-based model configuration with binary weights
• Batch Inference: Single and batched inference with optional parallelism
• Performance Tools: Built-in benchmarking and profiling utilities
• Comprehensive Testing: Full test coverage for all components

What's Inside

• Tensor abstraction with automatic memory management
• Polymorphic Layer hierarchy (Dense + ReLU)
• Sequential Model container for layer composition
• ModelLoader for JSON configs and binary weight files
• InferenceEngine with batching and profiling
• High-performance benchmark application
• Complete test suite

Quick Start

# Build the project
mkdir build && cd build
cmake ..
cmake --build . -j$(nproc)

# Run tests
ctest --output-on-failure

# Run benchmark with example model
./benchmark ../models/toy_mlp.json

Requirements

• C++ Compiler: GCC 7+ or Clang 5+ with C++17 support
• CMake: Version 3.14 or higher
• Threading: POSIX threads (included in most systems)
• Python 3: For generating model weights (optional)

Building from Source

1. Configure the build

mkdir build
cd build
cmake ..

2. Build all targets

cmake --build . -j$(nproc)

This builds:

• libmini_tensorrt.a - Static library
• benchmark - Benchmark application
• test_tensor, test_layers, test_model, test_engine - Test executables

3. Optional: Install

sudo cmake --install .

Running Tests

Run all tests with CTest:

cd build
ctest --output-on-failure

Or run individual test executables:

./test_tensor    # Core tensor operations
./test_layers    # Dense and ReLU layers
./test_model     # Model composition
./test_engine    # Inference engine

Expected output:

100% tests passed, 0 tests failed out of 4
Total Test time (real) = 0.04 sec

Usage

Running the Benchmark

Basic usage:

./benchmark ../models/toy_mlp.json

With custom parameters:

./benchmark ../models/toy_mlp.json \
  --warmup 10 \
  --iterations 100 \
  --batch 4 \
  --input-size 4

Options:

• --warmup <N> - Number of warmup iterations (default: 10)
• --iterations <N> - Benchmark iterations (default: 100)
• --batch <N> - Batch size for inference (default: 1)
• --input-size <N> - Input feature size (default: 4)

Example output:

=== Results ===
Total time: 0.307 ms
Average latency: 0.006 ms
Throughput: 162,866 inferences/sec

Using the Library in Your Code

#include "engine/inference_engine.hpp"
#include "model/model_loader.hpp"
#include "core/tensor.hpp"

using namespace mini_tensorrt;

int main() {
    // Load model from JSON
    auto model = model::ModelLoader::load_from_json("model.json");

    // Create inference engine
    engine::InferenceEngine engine(std::move(model));

    // Prepare input
    core::Tensor input(core::Shape({4}));
    input.data()[0] = 1.0f;
    input.data()[1] = 2.0f;
    input.data()[2] = 3.0f;
    input.data()[3] = 4.0f;

    // Run inference
    core::Tensor output = engine.infer(input);

    // Use output
    for (size_t i = 0; i < output.size(); ++i) {
        std::cout << output.at(i) << " ";
    }

    return 0;
}

Creating Custom Models

1. Define model architecture in JSON

{
  "input_dim": 4,
  "layers": [
    {
      "type": "dense",
      "in_features": 4,
      "out_features": 8,
      "weights_file": "layer1_weights.bin",
      "bias_file": "layer1_bias.bin"
    },
    {
      "type": "relu"
    },
    {
      "type": "dense",
      "in_features": 8,
      "out_features": 3,
      "weights_file": "layer2_weights.bin",
      "bias_file": "layer2_bias.bin"
    }
  ]
}

2. Generate weight files

Weights must be stored as binary files containing float32 values in row-major order:

import struct

# Example: Save weights for a 4x8 dense layer
weights = [0.1, 0.2, ...] # 32 values (4 * 8)
with open('layer1_weights.bin', 'wb') as f:
    for w in weights:
        f.write(struct.pack('f', w))

Use the provided script:

cd models
python3 generate_weights.py

Supported Features

• Layers: Dense (fully-connected), ReLU activation
• Inference: Single sample and batched inference
• Parallelism: Optional multi-threaded batch processing
• Model Format: JSON configuration with binary weights
• Data Type: Float32 (single precision)
• Memory: RAII-based automatic memory management
• Performance: High-resolution timing and profiling

Architecture

System Design

The project follows a modular architecture with clear separation of concerns:

┌─────────────────────────────────────┐
│      Benchmark Application          │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│      InferenceEngine                │  Single/Batch inference
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│      Model (Sequential)             │  Layer composition
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│      Layers (Dense, ReLU)           │  Forward pass
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│      Tensor & Shape                 │  Data storage
└─────────────────────────────────────┘

Core Components

Component	Description
Tensor	Float32 storage with RAII memory management
Shape	Dimension tracking and size calculation
Layer	Base class for neural network layers
DenseLayer	Fully-connected layer with matrix multiplication
ReLULayer	Element-wise activation function
Model	Sequential container for layer execution
ModelLoader	JSON parser and binary weight loader
InferenceEngine	High-level inference API with batching
Timer	High-resolution performance measurement
Profiler	Statistical profiling with aggregation

Project Structure

mini-tensorrt/
  CMakeLists.txt
  include/
    core/
      tensor.hpp
      shape.hpp
    layers/
      layer.hpp
      dense_layer.hpp
      relu_layer.hpp
    model/
      model.hpp
      model_loader.hpp
    engine/
      inference_engine.hpp
    utils/
      timer.hpp
      profiler.hpp
  src/
    core/
      tensor.cpp
      shape.cpp
    layers/
      dense_layer.cpp
      relu_layer.cpp
    model/
      model.cpp
      model_loader.cpp
    engine/
      inference_engine.cpp
    utils/
      timer.cpp
      profiler.cpp
  tests/
    test_tensor.cpp
    test_layers.cpp
    test_model.cpp
    test_engine.cpp
  apps/
    benchmark.cpp
  models/
    toy_mlp.json
    dense1_w.bin
    dense1_b.bin
    dense2_w.bin
    dense2_b.bin
  README.md

Technical Details

Memory Management

• RAII: Automatic resource management using smart pointers
• Contiguous Storage: Row-major layout for cache efficiency
• Zero-Copy: Efficient tensor operations without unnecessary copies

Performance Optimizations

• Row-major memory layout for better cache locality
• Batch-aware matrix multiplication
• Optional multi-threaded batch processing
• Efficient forward pass without dynamic allocations

Model Format

JSON configuration with separate binary weight files:

{
  "input_dim": 4,
  "layers": [
    {
      "type": "dense",
      "in_features": 4,
      "out_features": 8,
      "weights_file": "weights.bin",
      "bias_file": "bias.bin"
    },
    {
      "type": "relu"
    }
  ]
}

Weight Format:

• Binary files containing float32 values
• Row-major order for weight matrices
• Dense layer: [in_features × out_features] floats
• Bias: [out_features] floats

Development

Adding New Layers

1. Create header in include/layers/:

class MyLayer : public Layer {
public:
    std::string type() const override { return "mylayer"; }
    core::Tensor forward(const core::Tensor& input) override;
};

2. Implement in src/layers/
3. Register in ModelLoader::load_from_json()
4. Add tests in tests/test_layers.cpp

Code Organization

• Headers: include/ - Public API
• Implementation: src/ - Implementation files
• Tests: tests/ - Unit and integration tests
• Applications: apps/ - Executable programs
• Models: models/ - Example models and weights

Performance

Benchmark results on typical hardware:

Configuration	Latency (ms)	Throughput (inf/sec)
Single inference	0.006	~163,000
Batch=4	0.026	~155,000
Batch=8	0.048	~167,000

Results may vary based on CPU and model complexity

Limitations

• CPU-only: No GPU acceleration
• Layer types: Limited to Dense and ReLU
• Data type: Float32 only
• Memory: All data kept in RAM
• Execution: Sequential layer execution

Future Enhancements

Potential areas for expansion:

• Additional layer types (Conv2D, Pooling, Softmax)
• GPU support via CUDA or OpenCL
• Quantization (INT8, FP16)
• Model optimization and fusion
• ONNX model import
• Python bindings