Shake-Shake Regularization

A Chainer implementation of Shake-Shake regularization.

Results on CIFAR-10 (Shake-ResNet)

Model	Test Error (this implementation)	Test Error (in paper)
Shake-ResNet-26 2x32d (S-S-I)	Not tested yet	3.55 (average of 3 runs)
Shake-ResNet-26 2x64d (S-S-I)	2.85 (1 run)	2.98 (average of 3 runs)
Shake-ResNet-26 2x96d (S-S-I)	Not tested yet	2.86 (average of 5 runs)

• The model Shake-ResNet-26 2x64d (S-S-I) is trained with batch size 128, and initial learning rate 0.1.

Results on CIFAR-100 (Shake-ResNeXt)

Currently Shake-ResNeXt is not implemented.. Your contribution is more than welcome!

Dependency

• NVIDIA Driver
• NVIDIA Docker

Usage

0. Clone this repo

$ PROJ_DIR=~/shake_shake_chainer  # assuming you clone this repo to your home directory

$ git clone https://github.com/motokimura/shake_shake_chainer.git $PROJ_DIR

1. Build Docker image

Build docker image to setup the environment to train/test the model with Shake-Shake regularization.

$ cd $PROJ_DIR/docker
$ bash build.sh

2. Train Shake-ResNet model with CIFAR-10

Run docker container by following:

$ cd $PROJ_DIR/docker
$ bash run.sh

Now you should be inside the docker container you ran. Start training by following:

$(docker) cd /workspace
$(docker) python train_model.py

You can check training status and test accuracy from TensorBoard:

# Open another terminal window outside the container and type:
$ cd $PROJ_DIR/docker
$ bash exec.sh

# Now you should be inside the container already running. Start TensorBoard by following:
$(docker) tensorboard --logdir /workspace/results

Then, open http://localhost:6006 from your browser.

References

Papers

• Gastaldi, Xavier. "Shake-Shake regularization." [arXiv]
• Gastaldi, Xavier. "Shake-Shake regularization of 3-branch residual networks." [ICLR2017 Workshop]

Other implementations

• nutszebra's Chainer implementation. [GitHub]
• akitotakeki's Chainer implementation. [Gist]
• owruby's Pytorch implementation. [GitHub]
• Author's Lua implementation. [GitHub]

License

MIT License