First release

.github/workflows/ci-build.yml

@@ -63,7 +63,8 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install -r requirements.txt
+        pip install pybuilder
 
     - name: Run unit tests
       run: |
-        python -m unittest discover -s tests -p "test_*.py" -vvv
+        pyb -E unit

README.md

@@ -1,16 +1,129 @@
 # Lava DL
 
-__`lava-dl`__ is a library of deep learning tools, which consists of `lava.lib.dl.slayer` and `lava.lib.dl.netx` for training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.
+__`lava-dl`__ is a library of deep learning tools within Lava that support  offline training, online training and inference methods for various Deep Event-Based Networks.
 
-## Lava-dl Workflow
+There are two main strategies for training Deep Event-Based Networks: _direct training_ and _ANN to SNN converison_. 
+
+Directly training the network utilizes the information of precise timing of events. Direct training is very accurate and results in efficient networks. However, directly training networks take a lot of time and resources.
+
+On the other hand, ANN to SNN conversion is especially suitable for rate coded SNNs where we can leverage fast training of ANNs. These converted SNNs, however, typically require increased latency compared to directly trained SNNs.
+
+Lava-DL provides an improved version of [SLAYER](https://github.com/bamsumit/slayerPytorch) for direct training of deep event based networks and a new ANN-SNN accelerated training approach called [Bootstrap:TODO](link_here) to mitigate high latency issue of conventional ANN-SNN methods for training Deep Event-Based Networks.
+
+The lava-dl training libraries are independent of the core lava library since Lava Processes cannot be trained directly at this point. Instead, lava-dl is first used to train the model which can then be converted to a network of Lava processes using the netx library using platform independent hdf5 network description.
+
+The library presently consists of
+
+1. `lava.lib.dl.slayer` for natively training Deep Event-Based Networks.
+2. `lava.lib.dl.bootstrap` for training rate coded SNNs.
+
+Coming soon to the library
+1. `lava.lib.dl.netx` for training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.
+
+More tools will be added in the future.
+
+## Lava-DL Workflow
 
 <p align="center">
-<img src="https://user-images.githubusercontent.com/11490108/135362329-a6cf89e7-9d9e-42e5-9f33-102537463e63.png" alt="Drawing" style="max-height: 400px;"/>
+<img src="https://user-images.githubusercontent.com/29907126/140595634-a97886c6-280a-4771-830b-ae47a9324612.png" alt="Drawing" style="max-height: 400px;"/>
 </p>
 
+Typical Lava-DL workflow:
+* **Training:** using `lava.lib.dl.{slayer/bootstrap}` which results in a _hdf5 network description_. Training usually follows an iterative cycle of architecture design, hyperparameter tuning, and backpropagation training.
+* **Inference:** using `lava.lib.dl.netx` which generates lava proces from the hdf5 network description of the trained network and enables inference on different backends.
+
+## Installation
+
+### Cloning Lava-DL and Running from Source
+
+We highly recommend cloning the repository and using pybuilder to setup lava.
+ You will need to install pybuilder for the same.
+
+**Note:** We assume you have already setup Lava with virtual environment. Make sure `PYTHONPATH` contains path to Lava core library first.
+
+* Linux/MacOS: `echo $PYTHONPATH`
+* Windows: `echo %PYTHONPATH%`
+
+The output should contain something like this `/home/user/lava`
+
+#### [Linux/MacOS]
+```bash
+$ git clone git@github.com:lava-dl/lava.git
+$ cd lava-dl
+$ pip install -r build-requirements.txt
+$ pip install -r requirements.txt
+$ export PYTHONPATH=$PYTHONPATH:$(pwd)/src
+$ pyb -E unit
+```
+#### [Windows]
+```cmd
+cd %HOMEPATH%
+git clone git@github.com:lava-dl/lava.git
+cd lava-dl
+pip install -r build-requirements.txt
+pip install -r requirements.txt
+set PYTHONPATH=%PYTHONPATH%;%cd%\src
+pyb -E unit
+```
+
+You should expect the following output after running the unit tests:
+```
+PyBuilder version 0.13.3
+Build started at 2021-11-05 18:44:51
+------------------------------------------------------------
+[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
+[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
+[INFO]  Activated environments: unit
+[INFO]  Building lava-nc/lava-dl version 0.2.0
+......  PyBuilder Logs ...
+[INFO]  Running unit tests
+[INFO]  Executing unit tests from Python modules in /home/user/lava-dl/tests
+[INFO]  Executed 80 unit tests
+[INFO]  All unit tests passed.
+......  PyBuilder Logs ...
+------------------------------------------------------------
+BUILD SUCCESSFUL
+------------------------------------------------------------
+Build Summary
+             Project: lava-nc/lava-dl
+             Version: 0.2.0
+      Base directory: /home/user/lava-dl
+        Environments: unit
+               Tasks: prepare [45089 ms] analyze [660 ms] compile_sources [0 ms] run_unit_tests [184641 ms] package [1086 ms] run_integration_tests [0 ms] verify [0 ms] publish [15128 ms]
+Build finished at 2021-11-05 18:49:25
+Build took 273 seconds (273800 ms)
+```
+
+### [Alternative] Installing Lava from Binaries
+
+If you only need the lava package in your python environment, we will publish
+Lava releases via
+[GitHub Releases](https://github.com/lava-nc/lava-dl/releases). Please download
+the package and install it.
+
+Open a python terminal and run:
+
+#### [Windows/MacOS/Linux]
+```bash
+$ python3 -m venv python3_venv
+$ pip install -U pip
+$ pip install lava-nc-0.1.0.tar.gz
+```
+
+## Getting Started
+
+**End to end tutorials**
+* [Oxford spike train regression](dummy_link) TODO: UPDATE LINK
+* [MNIST digit classification](dummy_link) TODO: UPDATE LINK
+* [NMNIST digit classification](dummy_link) TODO: UPDATE LINK
+* [PilotNet steering angle prediction](dummy_link) TODO: UPDATE LINK
+
+**Deep dive tutorials**
+* [Dynamics and Neurons](dummy_link) TODO: UPDATE LINK
+
 ## __`lava.lib.dl.slayer`__ 
 
-`lava.lib.dl.slayer` is an enhanced version of [SLAYER](https://github.com/bamsumit/slayerPytorch). Most noteworthy enhancements are: support for _recurrent network structures_, a wider variety of _neuron models_ and _synaptic connections_ (a complete list of features is [here](https://github.com/lava-nc/lava-dl/blob/main/lib/dl/slayer/README.md)). This version of SLAYER is built on top of the [PyTorch](https://pytorch.org/) deep learning framework, similar to its predecessor. For smooth integration with Lava, `lava.lib.dl.slayer` supports exporting trained models using the platform independent __hdf5 network exchange__ format. 
+`lava.lib.dl.slayer` is an enhanced version of [SLAYER](https://github.com/bamsumit/slayerPytorch). Most noteworthy enhancements are: support for _recurrent network structures_, a wider variety of _neuron models_ and _synaptic connections_ (a complete list of features is [here_TODO:UPDATE](https://github.com/lava-nc/lava-dl/blob/main/lib/dl/slayer/README.md)). This version of SLAYER is built on top of the [PyTorch](https://pytorch.org/) deep learning framework, similar to its predecessor. For smooth integration with Lava, `lava.lib.dl.slayer` supports exporting trained models using the platform independent __hdf5 network exchange__ format. 
 
 In future versions, SLAYER will get completely integrated into Lava to train Lava Processes directly. This will eliminate the need for explicitly exporting and importing the trained networks. 
 
@@ -54,13 +167,80 @@ class Network(torch.nn.Module):
 __Training__
 ```python
 net = Network()
+assistant = slayer.utils.Assistant(net, error, optimizer, stats)
+...
+for epoch in range(epochs):
+    for i, (input, ground_truth) in enumerate(train_loader):
+        output = assistant.train(input, ground_truth)
+        ...
+    for i, (input, ground_truth) in enumerate(test_loader):
+        output = assistant.test(input, ground_truth)
+        ...
+```
+__Export the network__
+```python
+net.export_hdf5('network.net')
+```
+
+## __`lava.lib.dl.bootstrap`__
+
+In general ANN-SNN conversion methods for rate based SNN result in high latency of the network during inference. This is because the rate interpretation of a spiking neuron using ReLU acitvation unit breaks down for short inference times. As a result, the network requires many time steps per sample to achieve adequate inference results.
+
+`lava.lib.dl.bootstrap` enables rapid training of rate based SNNs by translating them to an equivalent dynamic ANN representation which leads to SNN performance close to the equivalent ANN and low latency inference. More details [here:TODO](link). It also supports _hybrid training_ with a mixed ANN-SNN network to minimize the ANN to SNN performance gap. This method is independent of the SNN model being used.
+
+It has similar API as `lava.lib.dl.slayer` and supports exporting trained models using the platform independent __hdf5 network exchange__ format.
+
+### Example Code
+
+__Import modules__
+```python
+import lava.lib.dl.bootstrap as bootstrap
+```
+__Network Description__
+```python
+# like any standard pyTorch network
+class Network(torch.nn.Module):
+    def __init__(self):
+        ...
+        self.blocks = torch.nn.ModuleList([# sequential network blocks 
+                bootstrap.block.cuba.Input(sdnn_params), 
+                bootstrap.block.cuba.Conv(sdnn_params,  3, 24, 3),
+                bootstrap.block.cuba.Conv(sdnn_params, 24, 36, 3),
+                bootstrap.block.cuba.Conv(rf_params, 36, 64, 3),
+                bootstrap.block.cuba.Conv(sdnn_cnn_params, 64, 64, 3),
+                bootstrap.block.cuba.Flatten(),
+                bootstrap.block.cuba.Dense(alif_params, 64*40, 100),
+                bootstrap.block.cuba.Dense(cuba_params, 100, 10),
+            ])
+
+    def forward(self, x, mode):
+        ...
+        for block, m in zip(self.blocks, mode):
+            x = block(x, mode=m)
+
+        return x
+
+    def export_hdf5(self, filename):
+        # network export to hdf5 format
+        h = h5py.File(filename, 'w')
+        layer = h.create_group('layer')
+        for i, b in enumerate(self.blocks):
+            b.export_hdf5(layer.create_group(f'{i}'))
+```
+__Training__
+```python
+net = Network()
+scheduler = bootstrap.routine.Scheduler()
 ...
 for epoch in range(epochs):
     for i, (input, ground_truth) in enumerate(train_loader):
-        out = net(input)
+        mode = scheduler.mode(epoch, i, net.training)
+        output = net.forward(input, mode)
         ...
+        loss.backward()
     for i, (input, ground_truth) in enumerate(test_loader):
-        out = net(input)
+        mode = scheduler.mode(epoch, i, net.training)
+        output = net.forward(input, mode)
         ...
 ```
 __Export the network__

build.py

@@ -22,7 +22,7 @@
 @init
 def set_properties(project):
     project.set_property("dir_source_main_python", "src")
-    project.set_property("dir_source_unittest_python", "tests")
+    project.set_property("dir_source_unittest_python", "tests/lava")
     project.set_property("dir_source_main_scripts", "scripts")
     project.set_property("dir_docs", "docs")
 
@@ -64,7 +64,7 @@ def set_properties(project):
 @init(environments="unit")
 def set_properties_unit(project):
     project.set_property("dir_source_main_python", "src")
-    project.set_property("dir_source_unittest_python", "tests")
+    project.set_property("dir_source_unittest_python", "tests/lava")
     project.set_property("dir_source_main_scripts", "scripts")
     project.set_property("dir_docs", "docs")
 

requirements.txt

@@ -0,0 +1,7 @@
+torch == 1.8.1
+torchvision == 0.9.1
+numpy
+scipy
+matplotlib
+ninja
+h5py >= 3.1.0

src/lava/lib/dl/bootstrap/README.md

@@ -0,0 +1,59 @@
+# Lava-DL Bootstrap
+
+Rate based SNNs generated by ANN-SNN conversion methods require long runtimes to achieve high accuracy because the rate-based approximation of the ReLU activation unit breaks down for short runtimes.
+
+`lava.lib.dl.bootstrap` accelerates rate coded Spiking Neural Network (SNN) training by dynamically estimating the equivalent ANN transfer function of a spiking layer with a picewise linear model at regular epochs interval and using the ANN equivlent network to train the original SNN. 
+
+**Highlight features**
+
+* Accelerated rate coded SNN training.
+* Low latency inference of trained SNN made possible by close modeling of equivalent ANN dynamics.
+* Hybrid training with a mix of SNN layers and ANN layers for minimal drop in SNN accuracy.
+* Scheduler for seamless switching between different bootstrap modes.
+
+## Bootstrap Training
+
+The underlying principle for ANN-SNN conversion is that the ReLU activation function (or similar form) approximates the firing rate of an LIF spiking neuron. Consequently, an ANN trained with ReLU activation can be mapped to an equivalent SNN with proper scaling of weights and thresholds. However, as the number of time-steps reduces, the alignment between ReLU activation and LIF spiking rate falls apart mainly due to the following two reasons (especially, for discrete-in-time models like Loihi’s CUBA LIF):
+
+![fit](https://user-images.githubusercontent.com/29907126/140595166-336e625d-c269-40d6-af85-caf5d2328139.png)
+
+* With less time steps, the SNN can assume only a few discrete firing rates.
+* Limited time steps mean that the spiking neuron activity rate often saturates to maximum allowable firing rate.
+
+In Bootstrap training. An SNN is used to jumpstart an equivalent ANN model which is then used to accelerate SNN training. There is no restriction on the type of spiking neuron or its reset behavior. Bootstrap training consists of the following steps:
+
+<p align="center">
+<img src="https://user-images.githubusercontent.com/29907126/140595174-2feb6946-bf64-4188-a6ea-eeb693a3052d.png" alt="Drawing" style="height: 400px;"/>
+</p>
+
+* Input output data points are first collected from the network running as an SNN: **`bootstrap.mode.SNN`**.
+* The data is used to estimate the corresponding ANN activation as a piecewise linear layer, unique to each layer: **``bootstrap.mode.FIT``** mode.
+* The training is accelerated using the piecewise linear ANN activation: **``bootstrap.mode.ANN``** mode.
+* The network is seamlessly translated to an SNN: **``bootstrap.mode.SNN``** mode.
+* SAMPLE mode and FIT mode are repeated for a few iterations every couple of epochs, thus maintaining an accurate ANN estimate.
+
+## Hybridization
+
+The dynamic estimation of the ANN activation function may still not be enough to reduce the gap between the SNN and it's equivalent ANN, especially for smaller inference timesteps and networks with increasing depth. In such a scenario, one can look at a hybrid approach of directly training a part of the network as SNN layers/blocks while acclearating the rest of the layers/blocks with bootstrap training.
+
+With `bootstrap.block` interface, some of the layers in the network can be run in SNN and rest in ANN. We define a **crossover** layer which splits layers earlier than it to always SNN and rest to ANN-SNN bootstrap mode.
+
+<p align="center">
+<img src="https://user-images.githubusercontent.com/29907126/140596065-e72e1340-351d-4e5f-b4e0-8b77ed95eb9a.png" alt="Drawing" style="height: 350px;"/>
+</p>
+
+## Tutorials
+
+* [MNIST digit classification](dummy_link) TODO: UPDATE LINK
+
+## Modules
+The main modules are 
+
+### `bootstrap.block`
+It provides `lava.lib.dl.slayer.block` based network definition interface.
+
+### `bootstrap.ann_sampler`
+It provides utilities for sampling SNN data points and pievewise linear ANN fit.
+
+### `bootstrap.routine`
+`bootstrap.routine.Scheduler` provides an easy scheduling utility to seamlessly switch between SAMPLING | FIT | ANN | SNN mode. It also provides ANN-SNN bootstrap **hybrid training** utility as well determined by crossover point.

src/lava/lib/dl/bootstrap/__init__.py

@@ -0,0 +1,8 @@
+# Copyright (C) 2021 Intel Corporation
+# SPDX-License-Identifier:  BSD-3-Clause
+
+
+from .block.base import Mode
+from . import block, ann_sampler, routine
+
+__all__ = ['block', 'ann_sampler', 'routine', 'Mode']