Why Horovod doesn't have compute and communication overlap when XLA is used?

[LiweiPeng]

Environment:

1. Framework: (TensorFlow, Keras, PyTorch, MXNet) Tensorflow
2. Framework version: 1.14
3. Horovod version: 0.16.4
4. MPI version: 3.1.1
5. CUDA version: 10.0
6. NCCL version: 2.4.7
7. Python version: 2.7.5
8. OS and version: CentOS 7.4
9. GCC version: 4.8.5

Checklist:

1. Did you search issues to find if somebody asked this question before? yes
2. If your question is about hang, did you read this doc?
3. If your question is about docker, did you read this doc?
4. Did you check if you question is answered in the troubleshooting guide?

Your question:
I used horovod to train distributed BERT model using nvidia's src code at https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT. I found when XLA is not used, horovod+bert has very good scalability (~1.92x from 1 to 16 GPUs). nvprof shows that there is good compute and communication (shown as 'mem' in the screenshot below) overlap, as expected.

NOTE: the screenshots below are nvprof results for a whole step.

However, when XLA is used, horovod+bert has much worse scalability (~1.8x from 1 to 16 GPUs). nvprof shows that there is little compute and communication overlap.

My question is: What caused this no compute and communication overlap when XLA is used?

[luomai]

@LiweiPeng Hi Liwei, did you resolve this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.