Why does Tensor-parallel Communication Overlap require MPI?

[mjkpolo]

Hello,
I was trying to replicate the MLCommons llama 2 70b training with h200s, but the cluster I am using doesn't support MPI. I got it working by setting

export TP_COMM_OVERLAP=0

and using a smaller model, because I noticed in the NeMo code:
https://github.com/NVIDIA/NeMo/blob/dc08eddab27ef1da0cbf4f272c25dedf7cb8f11e/nemo/lightning/_strategy_lib.py#L92
I am not very familiar with training or HPC applications, but why does this feature require MPI, and cannot use NCCL? I read this blog post to try and understand what tensor parallel communication overlap is, but I can't figure out why I need MPI for it.

Thanks!

[ashors1]

Hi, MPI is used by default to bootstrap the user buffers (see the TransformerEngine documentation here). However, NCCL bootstrap should also be supported now. You can try setting tp_comm_bootstrap_backend="nccl" here.