Reproducing GPipe accuracy results

[0xsamgreen]

Hi, thank you for creating this code. Have you been able to reproduce any of the GPipe accuracy claims with torchgpipe? If so, could you make that training code available?

[sublee]

Hi @SG2, thanks for your question.

We haven't reproduced the accuracy of any model due to focusing on performance and memory usage instead. But we have been evaluating the transparency of GPipe several times.

However, we are planning to compare accuracy with and without GPipe on a model, such as ResNet-101. I'll let you know when we prepared the result.

[sublee]

My coworker @wbaek has done the experiments for ResNet-101 Accuracy Benchmark. You can get the reproducible code in examples/resnet101_accuracy_benchmark.

[0xsamgreen]

Thanks @sublee and @wbaek!