Skip to content

imirzadeh/stable-continual-learning

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

Understanding the Role of Training Regimes in Continual Learning

Towards increasing stability of neural networks for continual learning (NeurIPS'20)

Note: I will add an updated version of the code soon. If you have problem reproducing the results, please see the instructions for reproducing experiment 1 and experiment 2.

1. Code Structure

The high level structure of the code is as follows:

root
├── stable_sgd
│   
└── external_libs
    └── continual_learning_algorithms
    └── hessian_eigenthings
  1. stable_sgd : implementations of our stable and plastic training regimen for SGD (in Pytorch).
  2. external_libs: third-party implementations we used for our experiments such as:
    2.1 continual_learning_algorithms Open source implementations for A-GEM, ER-Reservoir, and EWC (in Tensorflow).
    2.2 hessian_eigenthings: Open source implementation of deflated power iteration for eigenspectrum calculations (in Pytorch).

2. Setup & Installation

The code is tested on Python 3.6+, PyTorch 1.5.0, and Tensorflow 1.15.2. In addition, there are some other numerical and visualization libraries that are included in requirements.txt file. However, for convenience, we provide a script for setup:

bash setup_and_install.sh

3. Replicating the Results

Note: I will add an updated version of the code soon. If you have problem reproducing the results, please see the instructions for reproducing experiment 1 and experiment 2.

We provide scripts to replicate the results:

  • 3.1 Run bash replicate_experiment_1.sh for experiment 1 (stable vs plastic).
  • 3.2 Run bash replicate_experiment_2.sh for experiment 2 (Comparison with other methods with 20 tasks).
  • 3.3 Run bash replicate_appendix_c5.sh for the experiment in appendix C5 (Stabilizing other methods).

For faster replication, here we have only 3 runs per method per experiment, but we used 5 runs for the reported results.

About

Towards increasing stability of neural networks for continual learning: https://arxiv.org/abs/2006.06958.pdf (NeurIPS'20)

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published