gradient accumulation for stage 2

[eric-haibin-lin]

ZeRO stage 2 does not support gradient accumulation, if you need gradient accumulation please use stage 1

Stage 2 significantly reduces the memory footprint. Is there a plan to support gradient accumulation for stage 2? Thanks.

[ShadenSmith]

Thank you for you interest in DeepSpeed! We just added support for gradient accumulation in ZeRO-2. Please let us know if you run into any issues!

[gongjingcs]

hi, I see in ZeRO-2, communicates during non gradient accumulation boundaries as well

Is it for some reason? could you help me understand this?
Looking forward to your reply

[luojiapeng14]

I have the same question. why zero-2 need communicate gradient whether it is on the grad acc boundary?