Saving & loading optimizer state

[OverLordGoldDragon]

Optimizer weights are now stored in model.optimizer.optimizer, and model.optimizer.weights==[] - hence, default saving and loading methods will not work. Existing code can account for this as follows:

if "Lookahead" in str(model.optimizer):
    optimizer = model.optimizer.optimizer
else:
    optimizer = model.optimizer

optimizer.get_weights(...)
optimizer.set_weights(...)

NOTE: unsure if above accounts for all differences. Packing weights into model.optimizer directly will render this redundant.

[CyberZHG]

Fixed.

[OverLordGoldDragon]

@CyberZHG That was fast - thanks; I'll test it shortly. To clarify, is below

self.weights = self.optimizer.weights + slow_params

applying weight updates to self.weights, while Keras applies updates to self.optimizer.weights via self.updates? That is, if this is the only weight update, then before this fix, the 'slow' parameters weren't accounted for at all.

[OverLordGoldDragon]

Tested - model.optimizer.weights now holds weights, but also,

len(model.optimizer.updates) == len(model.optimizer.optimizer.updates) # 171 == 171
len(model.optimizer.weights) >  len(model.optimizer.optimizer.weights) # 137 > 103
# up to matching len, the two sets of weights are all equal

Is this intended? If so, was this accounted for before? It may explain my poor model performance. A snippet of the point where the weight tensors differ below:

---

(Also, optimizer save size has increased - unless the exact partial duplicate is necessary, better without it)

[OverLordGoldDragon]

Something's faulty - model performance plummets when loading states/weights and re-compiling model for a different batch_shape (greater timesteps); it behaves as if nothing was loaded, not even layer weights; I check that optimizer weights are loaded as saved

(val spikes should coincide with train spikes - plot error)

[OverLordGoldDragon]

Plummeting problem fixed; Lookahead appears strongly bound to its internal optimizer's state - the problem was solved by loading the iterations attribute, which before was thrown out as Nadam itself worked better that way. The iterations attr mediates Nadam's momentum - lower = lower, down to half of max. Same holds for abruptly changing .lr. Not rigorous evidence though, as still unsure whether Lookahead's properly implemented.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OverLordGoldDragon]

To clarify, is the weight len discrepancy mentioned in my third comment intentional or a bug?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.