🐫 camlgrad 📉

Inspired by micrograd and tinygrad, camlgrad is a toy autograd engine in OCaml from scratch using an Apple Accelerate backend for vectorized computation. camlgrad offers a Tensor module with a wide variety of unary operations and binary operations that can be composed in arbitrary ways to define computation graphs that admit forward and backward passes on scalar valued functions. Using the Tensor module, several other modules are defined for calculating losses (Loss), defining multi-layer perceptrons (Mlp) and performing gradient descent (Optimizer).

Installation

Install ocaml compiler and package manager (opam) along with some useful platform tools:

brew install ocaml
opam install ocaml-lsp-server odoc ocamlformat utop

Subsequently, we can create a switch (an isolated ocaml environment), activate it and install the packages necessary for camlgrad:

opam switch create camlgrad 5.1
opam switch camlgrad
opam install ocamlgraph ctypes-foreign alcotest

Usage

camlgrad makes it easy to define an MLP and perform forward and backward passes. As illustrated in the code snippet below, we can simply:

• Define an MLP containing a single layer with sigmoid activation
• Define arbitrary input and target
• Perform a forward pass on the MLP model
• Calculate binary cross entropy
• Update parameters of MLP using gradient descent with respect to the loss

let mlp = Mlp.get_mlp [|(Tensor.sigmoid, (100, 1))|] in
let input = Tensor.random (1, 100) in
let target = Tensor.ones (1, 1) in
let (pred, _) = Mlp.mlp_forward mlp input in 
let bce_loss = Loss.binary_cross_entropy pred target in
Optimizer.gradient_descent mlp bce_loss 0.01;

In order to visualize the computation graph, we can export a general specification of the graph into a file using:

Tensor.visualize_computation_graph bce_loss "graph.dot";

To render graph we use the Graphviz CLI:

dot -Tpng graph.dot -o graph.png

Note: Must first install graphviz with brew install graphviz

Apple Accelerate Details

Apple Accelerate is a set of API's to perform large-scale mathematical computations and image calculations, optimized for high performance and low energy consumption. The 2 APIs I leverage in particular are:

• BLAS: Perform common linear algebra operations with Apple’s implementation of the Basic Linear Algebra Subprograms (BLAS).
• VForce: Perform transcendental and trigonometric functions on vectors of any length.

The BLAS and VForce API are in C which we can easily interface with from OCaml using the Ctypes library. The Ctypes library lets you define the C interface in pure OCaml, and the library then takes care of loading the C symbols and invoking the foreign function call.

To represent tensors, the OCaml Bigarray is used. Bigrarray implements multi-dimensional arrays of integers and floating-point numbers. In particular, it allows efficient sharing of large numerical arrays between OCaml code and C or Fortran numerical libraries.

camlgrad only requires a few libraries outside the standard library (alcotest for testing, ctypes-foreign for foreign function interface and ocamlgraph for generating viusalization of computation graph).

Contributing

In the unlikely event someone read this far and is interested in contributing, feel free to put up an issue or pull request 😊