This is a reimplementation of Karpathy's scalar-valued autograd engine for micrograd as a Python C extension. There's about a 9x speedup overall during the training of a MLP on the two moons dataset. See the micrograd repository and Karpathy's backpropagation explanation video for more information. Because micrograd's engine is nicely self-contained and isolated from the Python neural network code, micrograd's engine is easily replaced with a compiled shared object file, as shown in the demo below:
This is micrograd's original engine in pure Python, about a 100 LOC Value engine class object:
engine.py.mp4
This is the Python C extension version, the Value engine reimplemented in C in about 500 LOC:
engine.c.mp4
- Clone https://github.com/karpathy/micrograd
- Build
engine.c:
pip install -r requirements.txt
python setup.py build_ext --inplace
python -m pytest test.py
- Replace https://github.com/karpathy/micrograd/blob/master/micrograd/engine.py with the built shared object file (as in the above demo)
I found a good introduction to the Python C API and the reference counting semantics to be Sam Gross's nogil talk. Sam Gross also makes a good point about using reversible debuggers (rr) which I found helpful, as well as Valgrind (while this extension works on any OS, Valgrind is Linux-only).
MIT