Neural Weight Search (NWS)

This is official implementation of Neural Weight Search for Task Incremental Learning.

NWS automatically builds new models by searching pretrained grouped weights saved in layer-wise pools and saves these models in the form of indices, which significantly reduces the memory cost.

NWS is a novel out-of-the-box mechanism that can be easily integrated with modern deep learning methods.

The experiments show NWS achieves state-of-the-art performance on the two task-incremental learning bencharmk, i.e., Split-CIFAR-100 and CUB-to-Sketches benchmarks in terms of both accuracy and memory.

Prerequisites

Create a new python env named nws

Way1: from requirements.txt

$ pip install -r requirements.txt

Way2: manually install packages

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install pytorch-lightning
pip install einops
pip install pytorch-lightning-bolts

Prepare Benchmark Datasets

Split-Cifar benchmark is available at here

Cub-to-Sketches benchmark has 5 fine-grained datasets which are available at here

Pretrained kernel pools and Experiment Results

Some of the results are listed in result_cls.md. We provide pretrained kernel pools on Imagenet in google drive

Quick Start

Download the ckpt folders or train your own kernel pools configs/xxx.yaml is the configuration file, which contains network architecture, NWS-related and training related parameters. scripts/xxx.sh is the script to run NWS experiments.

• Split CIFAR-100 experiments with NWS-incorporated incremental learning NOTE change the dataset path to your local machine

$ bash cifar_resnet18.sh

• CUB to Sketches experiments with NWS-incorporated incremental learning

$ bash cub2sketches_resnet18.sh

Advanced Usage for Slurm Cluster

scripts/job_launcher.sh is the script to submit NWS experiments on slurm cluster.

for example, to submit jobs for cifar_resnet18.sh with multiple random seeds:

python job_launcher.py -s cifar_res![img.png](img.png)net18.sh -n cifar_resn18 -a 2 

Known Issues

1.When pretaining the kernel pools, if amp16 is used, sometimes Nan loss (due to the overflow) may be induced.

Potential Applications

Potentially, the NWS can be applied to other ML domains like model compression, network sparsification.

We have also designed a toy experiment to show the effectiveness of NWS in network sparsification. The results can be found in the appendix of the paper.

Citation

If you found the provided code useful, please cite our work. Authors: Jian Jiang, Oya Celiktutan from King's College London

if you have any questions, please contact jian.jiang@kcl.ac.uk

Jiang, J., & Celiktutan, O. (2022). Neural Weight Search for Scalable Task Incremental Learning. __arXiv__. https://doi.org/10.48550/arXiv.2211.13823