Does DeepSpeed support tensor parallel for training?

[yiliu30]

I understand that Deepspeed supports TP for inference, but it does not support training.
Can someone share any documentation or demos if I'm incorrect?

[tjruwase]

You are correct, DeepSpeed relies on client-side tensor parallelism, such as Megatron, for training. The following doc provides some details on combining DeepSpeed and Megatron for this: https://huggingface.co/blog/bloom-megatron-deepspeed.

[yiliu30]

Thanks for your guides, it's a great resource to gain a broad understanding of these technologies.

[HeatherHong]

I have the same problem but i didnt find the ways that use only dp and tp automatically

[inkcherry]

Hi， @HeatherHong , TP+DP support for HFTrainer is supported, the doc will be added soon

[JaumePrats]

Any news on this matter?

[delock]

AutoTP training is now supported in DeepSpeed.
https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/huggingface-tp/README.md