irongrad

A deep learning framework built from scratch in C++17 with Python bindings via pybind11.

The long-term goal is not to be a smaller PyTorch. It is to be an inspectable deep learning framework: a system where tensors, computation graphs, gradients, optimizer updates, numerical issues, and kernel costs are first-class things users can see, explain, and modify.

irongrad is for people who want to understand what happens during training, not just call .backward() and trust the result. It should become a place to learn new deep learning ideas by implementing them directly: autograd, initialization, optimizers, normalization, attention, compilation, quantization, and whatever comes next.

Why I built it

PyTorch is excellent, but it can feel like a black box when learning. I wanted to implement the backward pass myself, from the chain rule, through the computation graph, down to raw memory, and understand what actually happens when you call .backward(). The C++ backend exists because that is where the interesting performance and memory decisions live.

The project is also meant to be a learning laboratory. Each feature should make the framework more capable, but also make the underlying idea easier to inspect. When irongrad implements a concept, the implementation should help answer questions like:

• What graph did this expression build?
• Which tensors kept history alive?
• Which gradients flowed, vanished, exploded, or stayed zero?
• Why did this optimizer update move a parameter by that amount?
• Which broadcasts, reductions, or shape transformations affected the backward pass?
• Where did numerical instability appear?
• What did a new technique actually change in the training dynamics?

That direction matters more than being lightweight. The point is to make training understandable and auditable.

Product direction

irongrad should grow into an inspectable framework for learning, debugging, and experimenting with deep learning internals.

Future APIs should make it natural to trace and explain a training step:

with irongrad.trace() as trace:
    y = model(x)
    loss = mse(y, target)
    loss.backward()
    opt.step()

trace.graph.show()
trace.gradients.inspect()
trace.updates.report()
trace.diagnose()

The framework should eventually expose:

• an autograd trace explorer that records ops, shapes, parents, outputs, and backward flow
• gradient health checks for zero, exploding, missing, NaN, and dead-activation gradients
• optimizer update audits showing parameter norms, gradient norms, and update ratios
• finite-difference gradient checks with clear failure reports
• operation explanations that connect formulas to actual tensors and shapes
• examples that teach concepts by running real code, not by hiding the mechanics

The strongest version of irongrad is a framework you use when standard frameworks are too opaque.

Quick example

from irongrad import Tensor

a = Tensor([1.0, 2.0, 3.0])
b = Tensor([4.0, 5.0, 6.0])

loss = (a * b).sum()
loss.backward()

print(a.grad)  # [4.0, 5.0, 6.0] — d(sum(a*b))/da = b
print(b.grad)  # [1.0, 2.0, 3.0] — d(sum(a*b))/db = a

Installation

# dev install (builds the C++ extension and installs editable)
make dev
make build

# or as a regular package
pip install .

Requires: Python 3.12+, CMake 3.18+, Ninja, a C++17 compiler.

Architecture

irongrad/
  tensor.py          Python Tensor wrapper
  nn/
    parameter.py     Trainable Tensor subclass
    layers.py        Layer definitions
    losses.py        Loss functions
    module.py        Base module class

include/irongrad/
  tensor.hpp         C++ Tensor, ops, graph links, and backward pass

src/
  bindings.cpp       pybind11 glue: add, mul, relu, sum, matmul

The Python frontend provides the user-facing API. It should stay thin: Python is for ergonomics, examples, reports, and high-level orchestration.

The core mechanics should live in a well-designed object-oriented C++ backend. Tensor storage, numerical updates, operation implementations, computation graph links, backward traversal, and future tracing metadata belong in C++. Over time, the backend should keep enough metadata to explain what happened during forward, backward, and optimizer steps.

Development

make dev       # uv sync + install pre-commit hooks
make build     # compile C++ extension and reinstall
make test      # pytest
make lint      # ruff + mypy
make format    # ruff format + clang-format
make clean     # remove build artifacts

C++ source lives in src/bindings.cpp and include/irongrad/. Pre-commit hooks run ruff, clang-format, and a few standard checks on every commit.

License

MIT