Difference between allgather and and allreduce

[base-y]

Hi, I am new to deepspeed. I came across the terms all_gather and all_reduce while syncing the gradients in multi-node/multi-gpu setting. It seems that both terms mean the same thing but not sure. Can I know what is the difference between those terms?

[tjruwase]

Please see torch's all_gather and all_reduce for descriptions.