can 3D parallelism example of bert or gpt-2 be provided?

[gongjingcs]

Hi,
I noticed DeepSpeed’s training engine has already provided hybrid data and pipeline parallelism and can be further combined with model parallelism such as Megatron-LM. can you provide a 3D parallelism example of bert or gpt-2?
Thanks

[ShivanshuPurohit]

You can find one here. I believe this is 3D as mp=2 and pp=2

[gongjingcs]

You can find one here. I believe this is 3D as mp=2 and pp=2

thanks alot！