How to make the recipe for fine-tuning Hifigan to work with grad_accumulation?

[bene-ges]

I try to finetune Hifigan using the recipe from NeMo tutorial.
It turns out to be incompatible with grad_accumulation.

Automatic gradient accumulation is not supported for manual optimization. Remove `Trainer(accumulate_grad_batches=5)` or switch to automatic optimization.

I suppose this happens because optim.sched is explicitly deleted when calling the script (like in the tutorial example).
So the question is - what to do if I still need to use grad_accumulation?
Are there any option combination to get similar constant learning behaviour without deleting the scheduler?