Saving and restoring models

[lucasb-eyer]

Also an important feature we should add soon-ish and I open an issue so we don't forget about it. And again, I think both of us have it in our toolboxes, so it should be easy.

[ikostrikov]

Do we need to add reseting of compiled functions when we restore models?

[lucasb-eyer]

Ah this question actually goes deeper than it appears :) I'd say, the feedforward and gradient accumulation don't need reset.

For the the optimizer's update function, it's tricky. Intuitively, I'd say the stored momentum and friends don't make sense anymore once we load different weights. Or maybe they do? I'm thinking of the "reload last best validation model and then decrease learning rate" use-case. Do we want to reset momentum etc. in that case, or not? Intuitively, I'd vote for yes, but I have the feeling most people don't do that, probably since they don't think that far. What's your vote?