[BERT/Tensorflow] Training speed on RTX 2080 ti

[kwonmha]

I'm doing pre-training of BERT and trying to find ways to speed up.
I just want to share and get checked the training speed on RTX 2080 ti of BERT.
Hope to know any way to improve speed though speed looks reasonable.

spec :
RTX 2080 ti(11GB memory)
tensorflow 1.14
sequence length = 512

n_gpu(w/ horovod)	fp32(batch 4)	amp+xla(batch 6)
1	13	32
2	18	43
4	59	77

the numbers in the table mean examples/sec

I wonder is there anything I can do more to speed up.
Unfortunately, due to large sequence length, I cannot increase batch size up to 8, which may enable to use TensorCore.
Also wanna compare with others.

[swethmandava]

I would recommend trying out gradient accumulation to emulate larger batches. It also provides a considerable speedup by delaying optimizer updates.

[kwonmha]

Thanks.
And what kind of reason would there be for that the speed doesn't increase in proportion to the number of gpu?

[kwonmha]

set batch_size 7 and the examples/sec grows to 86
set num_accumulation_steps 100 and the examples/sec grows to 89

[swethmandava]

How are you measuring examples/sec ? Is it the manual output from tf.estimator? It would help to get correct estimates once the model stabilizes since the first few updates are noisy and have significant overhead.

We usually see a < perfect linear scaling. It is strange that perf of gpu=4 fp32 / gpu =1 fp32 > 4..

[kwonmha]

Regarding the examples/sec on FP32, I agree with you.
I had tried to measure again but loss continued being Nan.

I'm more interested in training speed on FP16.

[swethmandava]

Based on scaling results (here)[https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT#fine-tuning-training-performance-for-squad-v11-on-dgx-1-16gb), going from 1 GPU to 4 GPU gives a ~3.62x speedup on DGX1 v16g. I see similar numbers on other GPUs as well. You seem to have a 2.4x speedup on your gpu - which is lower than expected but like I said, it could just be how you're measuring throughput. Feel free to use our scripts and methodology for a more apples-to-apples comparison.

[kwonmha]

I measured the speed by LogSessionRunHooks from estimator.

Thank you for your comments.