First release

README.md

@@ -1,16 +1,125 @@
 # Lava DL
 
-__`lava-dl`__ is a library of deep learning tools, which consists of `lava.lib.dl.slayer` and `lava.lib.dl.netx` for training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.
+__`lava-dl`__ is a library of deep learning tools within Lava that support  offline training, online training and inference methods for various Deep Event-Based Networks.
 
-## Lava-dl Workflow
+There are two main strategies for training Deep Event-Based Networks: _direct training_ and _ANN to SNN converison_. 
+
+Directly training the network utilizes the information of precise events in time. It is very accurate and results in efficient networks. However, the training these networks take a lot of time and resources.
+
+On the other hand, ANN to SNN conversion is especially suitable for rate coded SNNs where we can leverage the fast training of ANN. These converted SNNs, however, require increased latency compared to directly trained SNNs.
+
+Lava-DL provides improved [SLAYER](https://github.com/bamsumit/slayerPytorch) for direct training of deep event based networks and a new ANN-SNN accelerated training approach called [Bootstrap:TODO](link_here) to mitigate low latency issue of conventional ANN-SNN methods for training Deep Event-Based Networks.
+
+The Lava-DL training api's are independent of core Lava api since there is no native support for backpropagation in Lava processes. However, these training libraries support platform independent hdf5 network description protocol. Lava-DL also includes Network exchange library (coming soon...) that automatically generates the trained network using Lava processes and enables inference of the deep networks through lava on various backends.
+
+The library presently consists of
+
+1. `lava.lib.dl.slayer` for natively training Deep Event-Based Networks.
+2. `lava.lib.dl.bootstrap` for training rate coded SNNs.
+
+Coming soon to the library
+1. `lava.lib.dl.netx` for training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.
+
+More tools will be added in the future.
+
+## Lava-DL Workflow
 
 <p align="center">
-<img src="https://user-images.githubusercontent.com/11490108/135362329-a6cf89e7-9d9e-42e5-9f33-102537463e63.png" alt="Drawing" style="max-height: 400px;"/>
+<img src="https://user-images.githubusercontent.com/29907126/140595634-a97886c6-280a-4771-830b-ae47a9324612.png" alt="Drawing" style="max-height: 400px;"/>
 </p>
 
+Typical Lava-DL workflow consists of two parts:
+* **Training:** using `lava.lib.dl.{slayer/bootstrap}` which results in a _hdf5 network description_. Training usually consists of iterative cycle of architecture design, hyperparameter tuning, and backpropagation training.
+* **Inference:** using `lava.lib.dl.netx` which generates lava proces from the hdf5 network description of the trained network and enables inference on different backends.
+
+## Installation
+
+### Cloning Lava-DL and Running from Source
+
+We highly recommend cloning the repository and using pybuilder to setup lava.
+ You will need to install pybuilder for the same.
+
+**Note:** We assume you have already setup Lava with virtual environment.
+[TODO:verify on Windows and MacOS]
+
+#### [Linux/MacOS]
+```bash
+$ git clone git@github.com:lava-dl/lava.git
+$ cd lava-dl
+$ pip install -r build-requirements.txt
+$ pip install -r requirements.txt
+$ export PYTHONPATH=~/lava-dl #TODO change with directory restructure
+$ pyb -E unit
+```
+#### [Windows]
+```cmd
+cd %HOMEPATH%
+git clone git@github.com:lava-dl/lava.git
+cd lava
+pip install -r build-requirements.txt
+pip install -r requirements.txt
+set PYTHONPATH=%HOMEPATH%\lava #TODO change with directory resturcture
+pyb -E unit
+```
+
+You should expect the following output after running the unit tests:
+```
+PyBuilder version 0.13.3
+Build started at 2021-11-05 18:44:51
+------------------------------------------------------------
+[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
+[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
+[INFO]  Activated environments: unit
+[INFO]  Building lava-nc/lava-dl version 0.2.0
+......  PyBuilder Logs ...
+[INFO]  Running unit tests
+[INFO]  Executing unit tests from Python modules in /home/user/lava-dl/tests
+[INFO]  Executed 80 unit tests
+[INFO]  All unit tests passed.
+......  PyBuilder Logs ...
+------------------------------------------------------------
+BUILD SUCCESSFUL
+------------------------------------------------------------
+Build Summary
+             Project: lava-nc/lava-dl
+             Version: 0.2.0
+      Base directory: /home/user/lava-dl
+        Environments: unit
+               Tasks: prepare [45089 ms] analyze [660 ms] compile_sources [0 ms] run_unit_tests [184641 ms] package [1086 ms] run_integration_tests [0 ms] verify [0 ms] publish [15128 ms]
+Build finished at 2021-11-05 18:49:25
+Build took 273 seconds (273800 ms)
+```
+
+### [Alternative] Installing Lava from Binaries
+
+[TODO: verify] If you only need the lava package in your python environment, we will publish
+Lava releases via
+[GitHub Releases](https://github.com/lava-nc/lava-dl/releases). Please download
+the package and install it.
+
+Open a python terminal and run:
+
+#### [Windows/MacOS/Linux]
+```bash
+$ python3 -m venv python3_venv
+$ pip install -U pip
+$ pip install lava-nc-0.1.0.tar.gz
+```
+
+## Getting Started
+
+**End to end tutorials**
+* [Oxford spike train regression](dummy_link) TODO: UPDATE LINK
+* [MNIST digit classification](dummy_link) TODO: UPDATE LINK
+* [NMNIST digit classification](dummy_link) TODO: UPDATE LINK
+* [PilotNet steering angle prediction](dummy_link) TODO: UPDATE LINK
+
+**Deep dive tutorials**
+* [Dynamics and Neurons](dummy_link) TODO: UPDATE LINK
+
 ## __`lava.lib.dl.slayer`__ 
 
-`lava.lib.dl.slayer` is an enhanced version of [SLAYER](https://github.com/bamsumit/slayerPytorch). Most noteworthy enhancements are: support for _recurrent network structures_, a wider variety of _neuron models_ and _synaptic connections_ (a complete list of features is [here](https://github.com/lava-nc/lava-dl/blob/main/lib/dl/slayer/README.md)). This version of SLAYER is built on top of the [PyTorch](https://pytorch.org/) deep learning framework, similar to its predecessor. For smooth integration with Lava, `lava.lib.dl.slayer` supports exporting trained models using the platform independent __hdf5 network exchange__ format. 
+`lava.lib.dl.slayer` is an enhanced version of [SLAYER](https://github.com/bamsumit/slayerPytorch). Most noteworthy enhancements are: support for _recurrent network structures_, a wider variety of _neuron models_ and _synaptic connections_ (a complete list of features is [here_TODO:UPDATE](https://github.com/lava-nc/lava-dl/blob/main/lib/dl/slayer/README.md)). This version of SLAYER is built on top of the [PyTorch](https://pytorch.org/) deep learning framework, similar to its predecessor. For smooth integration with Lava, `lava.lib.dl.slayer` supports exporting trained models using the platform independent __hdf5 network exchange__ format. 
 
 In future versions, SLAYER will get completely integrated into Lava to train Lava Processes directly. This will eliminate the need for explicitly exporting and importing the trained networks. 
 
@@ -54,13 +163,80 @@ class Network(torch.nn.Module):
 __Training__
 ```python
 net = Network()
+assistant = slayer.utils.Assistant(net, error, optimizer, stats)
+...
+for epoch in range(epochs):
+    for i, (input, ground_truth) in enumerate(train_loader):
+        output = assistant.train(input, ground_truth)
+        ...
+    for i, (input, ground_truth) in enumerate(test_loader):
+        output = assistant.test(input, ground_truth)
+        ...
+```
+__Export the network__
+```python
+net.export_hdf5('network.net')
+```
+
+## __`lava.lib.dl.bootstrap`__
+
+In general ANN-SNN conversion methods for rate based SNN result in high latency of the network. This is because the rate interpretation of a spiking neuron using ReLU acitvation unit breaks down for short inference times. 
+
+`lava.lib.dl.bootstrap` enables rapid training of rate based SNNs by translating them to equivalent dynamic ANN representation which leads to SNN performance close to the equivalent ANN and low latency inference. More details [here:TODO](link). It also supports _hybrid training_ with ANN-SNN mixed network to minimize the ANN to SNN performance gap. This method is independent of the SNN model being used.
+
+It has similar API as `lava.lib.dl.slayer` and supports exporting trained models using the platform independent __hdf5 network exchange__ format.
+
+### Example Code
+
+__Import modules__
+```python
+import lava.lib.dl.bootstrap as bootstrap
+```
+__Network Description__
+```python
+# like any standard pyTorch network
+class Network(torch.nn.Module):
+    def __init__(self):
+        ...
+        self.blocks = torch.nn.ModuleList([# sequential network blocks 
+                bootstrap.block.cuba.Input(sdnn_params), 
+                bootstrap.block.cuba.Conv(sdnn_params,  3, 24, 3),
+                bootstrap.block.cuba.Conv(sdnn_params, 24, 36, 3),
+                bootstrap.block.cuba.Conv(rf_params, 36, 64, 3),
+                bootstrap.block.cuba.Conv(sdnn_cnn_params, 64, 64, 3),
+                bootstrap.block.cuba.Flatten(),
+                bootstrap.block.cuba.Dense(alif_params, 64*40, 100),
+                bootstrap.block.cuba.Dense(cuba_params, 100, 10),
+            ])
+
+    def forward(self, x, mode):
+        ...
+        for block, m in zip(self.blocks, mode):
+            x = block(x, mode=m)
+
+        return x
+
+    def export_hdf5(self, filename):
+        # network export to hdf5 format
+        h = h5py.File(filename, 'w')
+        layer = h.create_group('layer')
+        for i, b in enumerate(self.blocks):
+            b.export_hdf5(layer.create_group(f'{i}'))
+```
+__Training__
+```python
+net = Network()
+scheduler = bootstrap.routine.Scheduler()
 ...
 for epoch in range(epochs):
     for i, (input, ground_truth) in enumerate(train_loader):
-        out = net(input)
+        mode = scheduler.mode(epoch, i, net.training)
+        output = net.forward(input, mode)
         ...
+        loss.backward()
     for i, (input, ground_truth) in enumerate(test_loader):
-        out = net(input)
+        mode = scheduler.mode(epoch, i, net.training)
+        output = net.forward(input, mode)
         ...
 ```
 __Export the network__

lava/lib/dl/bootstrap/README.md

@@ -0,0 +1,59 @@
+# Lava-DL Bootstrap
+
+In general ANN-SNN conversion methods for rate based SNN result in high latency of the network. This is because the rate interpretation of a spiking neuron using ReLU acitvation unit breaks down for short inference times. 
+
+`lava.lib.dl.bootstrap` accelerates rate coded Spiking Neural Network (SNN) training by dynamically estimating the equivalent ANN transfer function of a spiking layer with a picewise linear model at regular interval and using the ANN equivlent network to train the original SNN. 
+
+**Highlight features**
+
+* Accelerated rate coded SNN training.
+* Low latency inference of trained SNN made possible by close modeling of equivalent ANN dynamics.
+* Hybrid training with a mix of SNN layers and ANN layers for minimal drop in SNN accuracy.
+* Scheduler for seamless switching between different bootstrap modes.
+
+## Bootstrap Training
+
+The underlying principle for ANN-SNN conversion is that the ReLU activation function (or similar form) approximates the firing rate of an LIF spiking neuron. Consequently, an ANN trained with ReLU activation can be mapped to an equivalent SNN with proper scaling of weights and thresholds. However, as the number of time-steps reduces, the alignment between ReLU activation and LIF spiking rate falls apart mainly due to the following two reasons (especially, for discrete-in-time models like Loihi’s CUBA LIF):
+
+![fit](https://user-images.githubusercontent.com/29907126/140595166-336e625d-c269-40d6-af85-caf5d2328139.png)
+
+* With less time steps, the SNN can assume only a few discrete firing rates.
+* Limited time steps mean that the spiking neuron activity rate often saturates to maximum allowable firing rate.
+
+In Bootstrap training. An SNN is used to jumpstart an equivalent ANN model which is then used to accelerate SNN training. There is no restriction on the type of spiking neuron or it's reset behavior. It consists of following steps:
+
+<p align="center">
+<img src="https://user-images.githubusercontent.com/29907126/140595174-2feb6946-bf64-4188-a6ea-eeb693a3052d.png" alt="Drawing" style="height: 400px;"/>
+</p>
+
+* Input output data points are first collected from the network running as an SNN: **`bootstrap.mode.SNN`**.
+* The data is used to estimate the corresponding ANN activation as a piecewise linear layer, unique to each layer: **``bootstrap.mode.FIT``** mode.
+* The training is accelerated using the piecewise linear ANN activation: **``bootstrap.mode.ANN``** mode.
+* The network is seamlessly translated to an SNN: **``bootstrap.mode.SNN``** mode.
+* SAMPLE mode and FIT mode are repeated for a few iterations every couple of epochs, thus maintaining an accurate ANN estimate.
+
+## Hybridization
+
+The dynamic estimation of ANN activation function may still not be enough to reduce the gap between SNN and it's equivalent ANN, especially when the inference timesteps are low and the networks grow deep. In such a scenario, one can look at a hybrid approach of directly training a part of the network as SNN layers/blocks while acclearating the rest of the layers/blocks with bootstrap training.
+
+With `bootstrap.block` interface, some of the layers in the network can be run in SNN and rest in ANN. We define **crossover** layer which splits layers earlier than it to always SNN and rest to ANN-SNN bootstrap mode.
+
+<p align="center">
+<img src="https://user-images.githubusercontent.com/29907126/140596065-e72e1340-351d-4e5f-b4e0-8b77ed95eb9a.png" alt="Drawing" style="height: 350px;"/>
+</p>
+
+## Tutorials
+
+* [MNIST digit classification](dummy_link) TODO: UPDATE LINK
+
+## Modules
+The main modules are 
+
+### `bootstrap.block`
+It provides `lava.lib.dl.slayer.block` based network definition interface.
+
+### `bootstrap.ann_sampler`
+It provides utilities for sampling SNN data points and pievewise linear ANN fit.
+
+### `bootstrap.routine`
+`bootstrap.routine.Scheduler` provides an easy scheduling utility to seamlessly switch between SAMPLING | FIT | ANN | SNN mode. It also provides ANN-SNN bootstrap **hybrid training** utility as well determined by crossover point.

lava/lib/dl/bootstrap/__init__.py

@@ -0,0 +1,8 @@
+# Copyright (C) 2021 Intel Corporation
+# SPDX-License-Identifier:  BSD-3-Clause
+
+
+from .block.base import Mode
+from . import block, ann_sampler, routine
+
+__all__ = ['block', 'ann_sampler', 'routine', 'Mode']