Is there a way to use optimize every n step?

[SimonLitak]

For example in pytorch one can store and aggregate losses in a variable which would allow to optimize after n seen batches.
A feature to Model's fit methods fit() and fit_generator() through a param "steps_to_optimize" like:

model.fit(x, y, batch_size=256, steps_to_optimize=4) => batchsize = 1024

so the Optimizer would store the losses for n seen batches, to optimize after n seen batches.

It would give the oppurtunity to optimize problems which need a big batch_size without being restricted to the memory of the device.
Of course for the fit() method there are problems like the overall number of steps has to be a multible of "steps_to_optimize" but at least in fit_generator() it should be doable.

[lvapeab]

Hi,

this conversation may be useful to you: #3556.

[SimonLitak]

Hey,
thanks so much, this is actually exactly what i was looking for.