-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' of github.com:tfjgeorge/nngeometry
- Loading branch information
Showing
1 changed file
with
9 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,25 @@ | ||
# NNGeometry | ||
|
||
[![Build Status](https://travis-ci.org/tfjgeorge/nngeometry.svg?branch=master)](https://travis-ci.org/tfjgeorge/nngeometry) | ||
|
||
NNGeometry allows you to: | ||
- compute **Fisher Information Matrices** (FIM) or derivates, using efficient approximatins such as low-rank matrices, KFAC, diagonal and so on | ||
- compute finite **Neural Tangent Kernels**, even for multiple output functions | ||
- easily and efficiently compute linear algebra operations involving these matrices **regardless of their approximation** | ||
|
||
Example: in the Elastic Weight Consolidation continual learning technique, you want to compute <img src="https://render.githubusercontent.com/render/math?math=\left(\mathbf{w}-\mathbf{w}_{A}\right)^{\top}F\left(\mathbf{w}-\mathbf{w}_{A}\right)">. It can be achieved with a block diagonal approximation for the FIM using: | ||
## Example | ||
|
||
In the Elastic Weight Consolidation continual learning technique, you want to compute <img src="https://render.githubusercontent.com/render/math?math=\left(\mathbf{w}-\mathbf{w}_{A}\right)^{\top}F\left(\mathbf{w}-\mathbf{w}_{A}\right)">. It can be achieved with a diagonal approximation for the FIM using: | ||
```python | ||
F = FIM(model=model, | ||
loader=loader, | ||
representation=PMatBlockDiag, | ||
representation=PMatDiag, | ||
n_output=10) | ||
|
||
regularizer = F.vTMv(w - w_a) | ||
``` | ||
If block diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatBlockDiag` to `PMatKFAC` in the above. | ||
If diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatDiag` to `PMatKFAC` in the above. | ||
|
||
## Documentation | ||
|
||
For more examples, you can visit the documentation at https://nngeometry.readthedocs.io |