Merge branch 'refactor'

.gitignore

@@ -1 +1,2 @@
 __pycache__
+*.swp

docs/api/generators.rst

@@ -0,0 +1,9 @@
+Generators
+==========
+
+The spirit of NNGeometry is that you do not directly manipulate Generator objects, but instead instantiate representations such as `PSpaceDense` or `PSpaceKFAC` so that you do not have to worry about implementing i.e. matrix-vector products for a `PSpaceKFAC` representation.
+
+.. automodule:: nngeometry.generator
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/index.rst

@@ -0,0 +1,21 @@
+API Reference
+=============
+
+.. warning::
+
+   This API reference is currently nothing but a dump of docstrings, ordered
+   alphabetically.
+
+The API reference contains detailed descriptions of the different end-user
+classes, functions, methods, etc. you will need to work with Blocks.
+
+.. note::
+
+   This API reference only contains *end-user* documentation. If you are
+   looking to hack away at Blocks' internals, you will find more detailed
+   comments in the source code.
+
+.. toctree::
+   :glob:
+
+   *

docs/api/pspace-representations.rst

@@ -0,0 +1,12 @@
+Parameter space matrix representations
+======================================
+
+.. autoclass:: nngeometry.object.pspace.PSpaceAbstract
+    :members:
+
+Concrete representations
+========================
+
+.. automodule:: nngeometry.object.pspace
+    :members:
+    :exclude-members: PSpaceAbstract

docs/api/vectors.rst

@@ -0,0 +1,15 @@
+Vector representations
+======================
+
+In NNGeometry, vectors are not just a bunch of scalars, but they have a semantic meaning.
+
+ - :class:`nngeometry.object.vector.PVector` objects are vectors living in the parameter space of a neural network
+ - :class:`nngeometry.object.vector.FVector` objects are vectors living in the function space of a neural network
+
+API documentation
+=================
+
+.. automodule:: nngeometry.object.vector
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/conf.py

@@ -0,0 +1,69 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'NNGeometry'
+copyright = '2020, Thomas George'
+author = 'Thomas George'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ['sphinx.ext.autodoc']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+import sphinx_rtd_theme
+html_theme = 'sphinx_rtd_theme'
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.doctest',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.todo',
+    'sphinx.ext.mathjax',
+    'sphinx.ext.graphviz',
+    'sphinx.ext.intersphinx',
+    # 'sphinx.ext.linkcode'
+    'torch'
+]
+
+master_doc = 'index'

docs/index.rst

@@ -0,0 +1,41 @@
+.. NNGeometry documentation master file, created by
+   sphinx-quickstart on Tue Nov 19 11:02:30 2019.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to NNGeometry's documentation!
+======================================
+
+NNGeometry is a library built on top of PyTorch aiming at giving tools
+to easily manipulate and study properties of Fisher Information Matrices and tangent kernels.
+
+You can start by looking at the quick start example below. Convinced? Then :doc:`install NNGeometry</install>`, try
+the tutorials or explore the API reference.
+
+.. warning::
+        NNGeometry is under developement, as such it is possible that core components change when
+        between versions.
+
+Tutorials 
+=========
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
+In-depth
+========
+
+.. toctree::
+   overview.rst
+   :maxdepth: 1
+
+   api/index.rst
+
+Quick start
+===========

docs/install.rst

@@ -0,0 +1,10 @@
+Installing NNGeometry
+=====================
+
+The development version of NNGeometry can be installed by directly cloning the ``master`` branch of the repository using ``pip``:
+
+.. code-block:: bash
+
+        $ pip install git+git://github.com/tfjgeorge/nngeometry.git
+
+The only dependency is PyTorch.

docs/overview.rst

@@ -0,0 +1,39 @@
+Overview
+========
+
+With NNGeometry, you can easily manipulate :math:`d \times d` matrices and :math:`d` vectors where :math:`d` is the number of parameter of your neural network, for modern neural networks where :math:`d` can be as big as :math:`10^8`. These matrices include for instance:
+
+ - The *Fisher Information Matrix* (FIM) used in statistics, in the natural gradient algorithm, or as an approximate of the Hessian matrix in some applications.
+ - *Posterior covariances* in Bayesian Deep Learning.
+
+You can also compute finite *tangent kernels*.
+
+A naive computation of the FIM would require storing :math:`d \times d` scalars in memory. This is prohibitively large for modern neural network architectures, and a line of research has focused at finding lower memory intensive approximations specific to neural networks, such as KFAC, EKFAC, low-rank approximations, etc. This library proposes a common interface for manipulating these different approximations, called *representations*.
+
+Let us now illustrate this by computing the FIM using the KFAC representation.
+
+   >>> F_kfac = FIM(layer_collection=layer_collection,
+                    model=model,
+                    loader=loader,
+                    representation=PSpaceKFAC,
+                    n_output=10,
+                    variant='classif_logits',
+                    device='cuda')
+   >>> print(F_kfac.trace())
+
+Computing the FIM requires the following arguments:
+
+ - The :class:`.layercollection.LayerCollection` ``layer_collection`` object describes the structure of the parameters that we will be manipulating. If we are interested in computing the FIM for the last 2 layers for instance, it will describe the structure of these last 2 layers and the size of their parameters.
+ - The :class:`torch.nn.Module` ``model`` object is the PyTorch model used as our neural network.
+ - The :class:`torch.utils.data.DataLoader` ``loader`` object is the dataloader that contains examples used for computing the FIM.
+ - The :class:`.object.pspace.PSpaceKFAC` argument specifies which representation to use in order to store the FIM.
+
+ We will next define a vector in parameter space, by using the current value given by our model:
+
+         >>> v = PVector.from_model(model)
+
+ We can now compute the matrix-vector product :math:`F v` by simply calling:
+
+        >>> Fv = F_kfac.mv(v)
+
+ Note that switching from the :class:`.object.pspace.PSpaceKFAC` representation to any other representation such as :class:`.object.pspace.PSpaceDense` is as simple as passing ``representation=PSpaceDense`` when building the ``F_kfac`` object.