Skip to content

A minimal autograd engine and neural network library for machine learning students.

Notifications You must be signed in to change notification settings

danielvegamyhre/autograd-for-dummies

Repository files navigation

autograd-for-dummies

build

A minimal autograd engine and neural network library written from scratch. Inspired heavily by Andrej Karpathy's micrograd, but has lots of comments explaining the math, concepts etc. along the way.

Autograd Engine

autograd.engine module implements a scalar valued autograd engine, which tracks values, their gradients, and the executed operations (and the resulting new values) in the form of a DAG (directed acyclical graph).

In this DAG the leaves are the input values and the roots are the outputs of the computational graph.

By tracing this graph from root to leaves, you can calculate the gradient of each node by using the chain rule.

This is called backpropagation, or more formally, reverse-mode automatic differentiation.

Example with Computational Graph

Create a Scalar.

a = Scalar(1.5)

Do some calculations.

b = Scalar(-4.0)
c = a**3 / 5
d = c + (b**2).relu()

Compute the gradients via reverse-mode automatic differentiation.

d.backward()

Display the computational graph with graphviz.

draw_graph(d)

Neural Network Library

Example: Simple Neural Network (Feed-Forward Multilayer Perceptron)

autograd.nn contains a simple neural network library, which contains building blocks for simple neural nets such as neurons, layers, and even a MLP (multi-layer perceptron).

autograd.utils contains some important utilities such as common loss functions, regularization techniques, and more.

(specifically a feed forward MLP) binary classifier, and achieve 100% accuracy on the classic moon dataset.

The notebook demo_nn.ipynb demonstrates how to use the neural network library to build and train a simple neural net (specifically a feed forward MLP) binary classifier, and achieve 100% accuracy on the classic moon dataset.

About

A minimal autograd engine and neural network library for machine learning students.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published