Skip to content

Commit

Permalink
Merge branch 'master' of github.com:tfjgeorge/nngeometry
Browse files Browse the repository at this point in the history
  • Loading branch information
tfjgeorge committed Sep 16, 2020
2 parents fd887d7 + ad9ffcf commit 86e9ab8
Showing 1 changed file with 9 additions and 3 deletions.
12 changes: 9 additions & 3 deletions Readme.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,25 @@
# NNGeometry

[![Build Status](https://travis-ci.org/tfjgeorge/nngeometry.svg?branch=master)](https://travis-ci.org/tfjgeorge/nngeometry)

NNGeometry allows you to:
- compute **Fisher Information Matrices** (FIM) or derivates, using efficient approximatins such as low-rank matrices, KFAC, diagonal and so on
- compute finite **Neural Tangent Kernels**, even for multiple output functions
- easily and efficiently compute linear algebra operations involving these matrices **regardless of their approximation**

Example: in the Elastic Weight Consolidation continual learning technique, you want to compute <img src="https://render.githubusercontent.com/render/math?math=\left(\mathbf{w}-\mathbf{w}_{A}\right)^{\top}F\left(\mathbf{w}-\mathbf{w}_{A}\right)">. It can be achieved with a block diagonal approximation for the FIM using:
## Example

In the Elastic Weight Consolidation continual learning technique, you want to compute <img src="https://render.githubusercontent.com/render/math?math=\left(\mathbf{w}-\mathbf{w}_{A}\right)^{\top}F\left(\mathbf{w}-\mathbf{w}_{A}\right)">. It can be achieved with a diagonal approximation for the FIM using:
```python
F = FIM(model=model,
loader=loader,
representation=PMatBlockDiag,
representation=PMatDiag,
n_output=10)

regularizer = F.vTMv(w - w_a)
```
If block diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatBlockDiag` to `PMatKFAC` in the above.
If diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatDiag` to `PMatKFAC` in the above.

## Documentation

For more examples, you can visit the documentation at https://nngeometry.readthedocs.io

0 comments on commit 86e9ab8

Please sign in to comment.