This is the code to training a DenseNet on CIFAR10 with meta-learned learning rates via higher.
We assume you have access to a gpu that can run CUDA 9.2. Then, the simplest way to install all required dependencies is to create an anaconda environment by running:
conda env create -f conda_env.yml
After the instalation ends you can activate your environment with:
source activate pytorch_env
To start the training script, simply run:
python train.py \
--work_dir ./log \
--meta_batch_size 16 \
--meta_num_train_steps 5 \
--meta_num_test_steps 5 \
--split_ratio 0.01
This will produce 'log' folder, where all the outputs are going to be stored including train/test/meta logs, and tensorboard blobs. One can attach a tensorboard to monitor training by running
tensorboard --logdir log
and opening up tensorboad in your browser. The console output is also available in a form:
| train | T: 100 | E: 1 | D: 11.1 s | L: 2.0694 | A: 24.1094 | LR: 0.0000
a training entry decodes as:
train - training episode
T - total updates
E - total number of epochs
D - duration
L - loss
A - accuracy
There will be similar entries from test and meta iterations. Below is the accuracy compasison for three runs: fixed learning rate, manually annealed, and learned with higher.
