Scatter-reduce in stage 2

[ghadialhajj]

The way I understand gradient partitioning is that during the backward pass, after each GPU computes the gradients for a given layer on its own batch, only one GPU (the one holding the optimizer state for that layer) would do a reduce operation to get the gradients from all batches.

In the paper, however, it says, when describing stage 2: "ZeRO only requires a scatter-reduce operation on the gradients". First is this "scatter-reduce" the same as the one here? If so, why do we need that instead of a simple reduce?

Thanks :)