Fetching contributors…
Cannot retrieve contributors at this time
140 lines (112 sloc) 6.86 KB

# The `ad` module

The `ad` module implements reverse-mode automatic differentiation. Computations using AD functions implicitly build a graph of all operations. When the computation finishes, the graph can be walked backward to accumulate partial derivatives from the output back to the inputs.

There is a simple interface for interacting with AD functions and AD values:

```var ad = require('adnn/ad');

// Raw Numbers/Tensors can be used with AD functions
ad.tensor.tanh(new Tensor([3]).fill(1.5));  // [0.9051, 0.9051, 0.9051]

// To compute derivatives, we must first turn input Numbers/Tensors into AD graph nodes
//    by 'lifting' them

// Feeding these nodes into AD functions results in Node outputs, which can be used to
//    initialize backpropagation
scalarOut.backprop();

// We can then retrieve the values and derivatives of different nodes

// It's also possible to check whether a value is a lifted AD Node or not

• Unary operators
• Defined for both scalars (in `ad.scalar`) and tensors (in `ad.tensor`)
• floor, ceil, round, sqrt, exp, log, abs, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, sigmoid
• Binary operators
• Defined for both scalars (in `ad.scalar`) and tensors (in `ad.tensor`). Tensor operators can accept a scalar as their second argument.
• add, sub, mul, div, pow, min, max, atan2
• Comparators
• Currently defined for scalars only (in `ad.scalar`)
• eq (==), neq (!=), peq (===), pneq (!==), gt (>), lt (<), geq (>=), leq (<=)
• Reductions
• `ad.scalar.sum(lst)`: Returns the sum of the list of scalars `lst`.
• `ad.tensor.sumreduce(x)`: Returns the sum of entries of a tensor `x`.
• `ad.tensor.allreduce(x)`: Returns true if all entries of a tensor `x` are nonzero.
• `ad.tensor.anyreduce(x)`: Returns true if any entry of a tensor `x` is nonzero.
• Indexing and Reshaping
• `ad.tensor.get(x, i)`: Extracts the `i`th element of `x` and returns it as a scalar.
• `ad.tensor.toScalars(x)`: Turns tensor `x` into an array of scalars.
• `ad.tensor.fromScalars(lst)`: Turns a list of scalars `lst` into a tensor.
• `ad.tensor.range(x, i, j)`: Returns a tensor constructed from elements `i` through `j` (non-inclusive) of tensor `x`.
• `ad.tensor.split(x, sizes)`: Split tensor `x` into `sizes.length` tensors, where the size of the output tensors are given by `sizes`.
• `ad.tensor.concat(lst)`: Concatenate a list of tensors `lst` into one tensor.
• Linear Algebra
• Requires inputs to be rank 2 tensors (i.e. matrices are NxN, vectors are Nx1).
• `ad.tensor.transpose(x)`: Returns the transpose of the matrix `x`.
• `ad.tensor.diagonal(x)`: Returns a diagonal matrix whose diagonal entries are the entries of the vector `x`.
• `ad.tensor.inverse(x)`: Returns the inverse of the matrix `x`.
• `ad.tensor.determinant(x)`: Returns the determinant of the matrix `x`.
• `ad.tensor.dot(x, y)`: Returns the inner product of the matrices `x` and `y`.
• `ad.tensor.cholesky(x)`: Returns the Cholesky decomposition of the matrix `x`.
• Miscellaneous
• `ad.tensor.softmax(x)`: Compute the Softmax function for a tensor `x`.

### Defining new primitives

```var ad = require('adnn/ad');

// Defining unary functions
OutputType: // Either Number or Tensor
name:       // The name of the new function
forward: function(x) {...}  // Implements the function
backward: function(xnode) {...} // Accumulates into derivative(xnode). Output node available as 'this'
});

// Defining binary functions
OutputType: // Either Number or Tensor
name:       // The name of the new function
forward: function(x, y) {...}  // Implements the function
backward1: function(xnode, y) {...} // Accumulates into derivative(xnode). Output node available as 'this'
backward2: function(x, ynode) {...} // Accumulates into derivative(ynode). Output node available as 'this'
});

// Defining arbitrary functions
OutputType: // Either Number or Tensor
name:       // The name of the new function
forward: function(...) {...}  // Implements the function
backward: function(...) {...} // Accumulates into derivatives of all Node inputs. Output node available as 'this'
getParents: function(...) {...} // Returns a list of inputs which are Nodes.
});

// Can also 'lift' non-differentiable functions to operate on Nodes
var eq = ad.liftBinaryFunction(function(x, y) { return x == y; });```

### Macro transforms for scalar code

Code which uses scalar math functions can be automatically converted to use scalar AD functions via a Sweet.js macro transform (see macros.sjs). There are several different ways to accomplish this:

#### Using node via command line / REPL

In this setting, the simplest way to use the macro transform is via the `ad.macroRequire` function:

```// In a file called 'dist.js':
function dist(x1, y1, x2, y2) {
var xdiff = x1 - x2;
var ydiff = y1 - y2;
return Math.sqrt(xdiff*xdiff + ydiff*ydiff);
}
module.exports = dist;

// -------------------------------------

// In a separate file:
Currently, `ad.macroRequire` is not available in the browser, as attempting to load a browserified script which includes Sweet.js will throw an error. For the time being, one workaround is to directly use Sweet.js to macro-transform your code (see their FAQ section on How to run Sweet.js in the browser).