Feature Request: add an additional argument to auto_optim to allow for gradient accumulation

[sandylaker]

When we use horovod backend and perform gradient accumulation, we get the following error:
AssertionError: Gradients were computed more than backward_passes_per_step times before call to step(). Increase backward_passes_per_step to accumulate gradients locally.

Thus, we need to change the default argument backward_passes_per_step of horovod.DistributedOptimizer to enable gradient accumulation in the distributed setting. To do so, we can add this argument to ignite.distributed.auto_optim.

[vfdev-5]

@sandylaker thanks for the feature request ! Maybe, we can enable kwargs for auto_optim as it is done for auto_model. In the docs we can explicitly say where kwargs goes exactly.

[Chandan-h-509]

Hey @vfdev-5 .
I would like to work on this issue.
Before I start I just want to know whether I have to add backward_passes_per_step as an argument into the ignite.distributed.auto_optim function?

[trsvchn]

Hey @Chandan-h-509 !
The idea is to introduce kwargs argument for the auto_optim and pass it to DistributedOptimizer like we did for the auto_model in a very similar way.
And then we need to describe in a doc very carefully, where it (kwargs) goes exactly.

[Chandan-h-509]

ohk.. Got it.. Can i work on it?

[trsvchn]

Sure! Go ahead! Thanks!