-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
14 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,15 +1,19 @@ | ||
# NNGeometry | ||
|
||
Explore the geometry of the function spaces described by Pytorch nn.Module functions with large d x d matrices where d is the number of parameters, using different sparse representations such as low-rank matrices, KFAC, diagonal matrix, and so on. | ||
NNGeometry allows you to: | ||
- compute **Fisher Information Matrices** (FIM) or derivates, using efficient approximatins such as low-rank matrices, KFAC, diagonal and so on | ||
- compute finite **Neural Tangent Kernels**, even for multiple output functions | ||
- easily and efficiently compute linear algebra operations involving these matrices **regardless of their approximation** | ||
|
||
# Documentation | ||
|
||
The documentation is available at https://nngeometry.readthedocs.io | ||
|
||
## Installation of the current repository | ||
Example: in the Elastic Weight Consolidation continual learning technique, you want to compute <img src="https://render.githubusercontent.com/render/math?math=\left(\mathbf{w}-\mathbf{w}_{A}\right)^{\top}F\left(\mathbf{w}-\mathbf{w}_{A}\right)">. It can be achieved with a block diagonal approximation for the FIM using: | ||
```python | ||
F = FIM(model=model, | ||
loader=loader, | ||
representation=PMatBlockDiag, | ||
n_output=10) | ||
|
||
regularizer = F.vTMv(w - w_a) | ||
``` | ||
git clone https://github.com/tfjgeorge/nngeometry.git | ||
cd nngeometry | ||
pip install . | ||
``` | ||
If block diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatBlockDiag` to `PMatKFAC` in the above. | ||
|
||
For more examples, you can visit the documentation at https://nngeometry.readthedocs.io |