Detailed description of 3D parallel operation mechanism

[gongjingcs]

hi，I noticed currently deepspeed has already supported pipeline parallelism. I am a little confused about its operating mechanism. Can you provide a detailed description diagram or operating mechanism description of it?
This will be helpful for us to fully understand pipeline parallelism strengths. Thanks alot

[samyam]

@gongjingcs have you had a chance to take a look at the following blog from our team:

https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/

We have tried to offer the details here.

[gongjingcs]

@gongjingcs have you had a chance to take a look at the following blog from our team:

https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/

We have tried to offer the details here.

Hello, thank you for your guidance. After reading this article, I still have a little question about why the pipeline has to perform gradient accumulation for n steps before completing an allreduce communication.