Skip to content
No description, website, or topics provided.
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md Update README.md Oct 4, 2019
__init__.py u Sep 10, 2019
acc.py u Oct 10, 2019
imagenet_input.py u Sep 10, 2019
imagenet_input.pyc u Oct 10, 2019
log.out u Sep 11, 2019
optimization.py u Oct 10, 2019
optimization.pyc u Oct 10, 2019
print.py u Sep 10, 2019
resnet_main.py u Oct 10, 2019
resnet_model.py u Sep 10, 2019
resnet_model.pyc u Oct 10, 2019
resnet_preprocessing.py u Sep 10, 2019
resnet_preprocessing.pyc u Oct 10, 2019
run.sh u Oct 10, 2019

README.md

How to reproduce?

Just hit ./run.sh

#################################################################################################

Please check log.out for the training log.

#################################################################################################

This is an implementation of LAMB optimizer by TensorFlow for ImageNet/ResNet-50 training.

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

https://arxiv.org/pdf/1904.00962.pdf

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh

#################################################################################################

This implementation can get 76.3% accuracy for ImageNet/ResNet-50 training in just 3519 iterations (batch size = 32K).

State-of-the-art optimizer like Adam fails to achieve this level of accuracy for large-batch training.

The authors significantly tuned the hyper-parameters of Adam in https://arxiv.org/pdf/1904.00962.pdf

#################################################################################################

We use 128 v3 TPU chips in this experiment. Because of the distributed batch normalization, the accuracy will be higher if you increase the number of chips to 256 or 512.

#################################################################################################

You can’t perform that action at this time.