micrograd_pybind_cpp

A minimal scalar-valued autograd engine implemented in C++ and exposed to Python via pybind11. Inspired by Andrej Karpathy's micrograd.

Overview

The engine builds a dynamic computation graph as arithmetic operations are performed on Value nodes. Calling .backward() on the output node propagates gradients back through the graph using reverse-mode automatic differentiation (backpropagation).

Project Structure

value.h          # Core C++ Value struct with forward ops and backward pass
bindings.cpp     # pybind11 bindings exposing Value to Python
setup.py         # Build script (distutils/setuptools extension)
requirements.txt # Python dependencies (pybind11, setuptools)
backprop_test.py # Example usage / sanity check

How It Works

Value (value.h)

Each Value wraps a scalar double and carries:

Field	Purpose
data	The forward-pass scalar value
grad	Accumulated gradient (populated by .backward())
prev	Child nodes that produced this value
_backward	Closure that computes local gradients for this op

Operations create a new output Value, store the inputs in prev, and capture a lambda in _backward that applies the chain rule:

• add — $\frac{\partial L}{\partial a} \mathrel{+}= \frac{\partial L}{\partial \text{out}}$, same for $b$
• mul — $\frac{\partial L}{\partial a} \mathrel{+}= b \cdot \frac{\partial L}{\partial \text{out}}$, $\frac{\partial L}{\partial b} \mathrel{+}= a \cdot \frac{\partial L}{\partial \text{out}}$

backward() performs a topological sort of the computation graph (DFS via build_topo), seeds this->grad = 1.0, then iterates the sorted nodes in reverse order, calling each node's _backward closure.

Memory is managed entirely through std::shared_ptr<Value>, and std::enable_shared_from_this is used so nodes can safely capture themselves inside lambdas.

bindings.cpp

Registers the Value class with pybind11 as the micrograd_cpp Python module. Exposes:

• Constructor: Value(float)
• Attributes: .data, .grad
• Methods: .add(), .mul(), .backward()
• Python operators: +, *

Build & Install

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Build and install the extension in-place
python setup.py build_ext --inplace

Usage

import micrograd_cpp as mg

a = mg.Value(2.0)
b = mg.Value(3.0)

c = a * b + a   # c = a*b + a  =>  c = 8.0

c.backward()

print(a.grad)   # 4.0  (dc/da = b + 1 = 3 + 1)
print(b.grad)   # 2.0  (dc/db = a = 2)

Worked Example

Given $c = a \cdot b + a$ with $a = 2,\ b = 3$:

$$\frac{\partial c}{\partial a} = b + 1 = 4 \qquad \frac{\partial c}{\partial b} = a = 2$$

Running backprop_test.py prints exactly these values, confirming the gradients are correct.

Limitations

• Only + and * are implemented; no activation functions (tanh, ReLU, etc.)
• No support for in-place operations or scalar–Value mixed arithmetic beyond what Python's operator dispatch handles automatically

Future Scope

The following features are missing from the current implementation and will be added in future updates:

• More ops — subtraction (-), division (/), power (**), and negation (__neg__)
• Activation functions — tanh, ReLU, sigmoid, exp, log
• Mixed scalar arithmetic — allow expressions like a + 2.0 or 3.0 * a without manually wrapping scalars in Value
• Neuron / Layer / MLP classes — higher-level neural network building blocks on top of Value, mirroring the full micrograd API
• Loss functions — mean squared error, cross-entropy
• Zero-grad utility — a .zero_grad() method to reset all gradients before each training step
• op label tracking — store the operation name on each node to enable computation graph visualization
• Graph visualization — Graphviz-based rendering of the forward computation graph with node data and grad values
• PyPI packaging — pyproject.toml / setup.cfg migration and installable wheel distribution