add gradient normalization and accumulation in supervised_training_step_* functions

[ydcjeff]

🚀 Feature

After #1589 , grad_norm is left to add (#419), and also gradient accumulation is also common in supervised training, we should also add that to have full supervised training.

[vfdev-5]

@ydcjeff yes, adding a grad accumulation could a be great addition ! We provide a code snippet in FAQ for that but in the code it could be also helpful.

[KickItLikeShika]

Hey @ydcjeff can i work on this?

[ydcjeff]

Hi @KickItLikeShika Sure, feel free to go ahead

[Ishan-Kumar2]

Hi @ydcjeff and @vfdev-5 I would like to add gradient accumulation in the create_supervised_trainer. It can be added as a seperate step like supervised_training_step_grad_acc which applies just grad acc or would it be useful to allow grad acc as a flag in the existing supervised_training_step* functions so that it can be coupled with existing ones like amp?

[vfdev-5]

@Ishan-Kumar2 I think a flag to existing supervised_training_step* can be a good option.