Skip to content

fasterai 0.4.1

Latest

Choose a tag to compare

@nathanhubens nathanhubens released this 16 Sep 14:42
· 6 commits to master since this release
435491a

Bug Fixes

  • PruneCallback now re-points learn.opt at the model's live parameters after every prune. Structured pruning replaces the nn.Parameter objects of the layers it shrinks, and the optimizer Learner.fit built still referenced the old ones: on a small convolutional model the optimizer held 1 of the model's 10 parameters after the first prune, and the pruned layers' weights did not change again for the rest of the fit (max |ΔW| = 0.0 over the following epoch). The optimizer object itself is kept and its parameter groups are rewritten in place, so the groups, the hypers and the state of this fit survive the prune, and fastai's no-weight-decay and force-train marks are re-applied. Frozen groups stay frozen: torch-pruning re-creates the parameters it replaces requiring grad, including those of a layer it touches only as a dependency of the one it prunes, so a frozen layer used to start training again after the first prune
  • The optimizer state of a parameter the prune replaced is now carried over instead of dropped: every state tensor shaped like the old parameter is sliced exactly as the parameter was (momentum, second moments), while scalars such as the step counter are kept, so the moments keep their history and the bias correction goes on. Every replacement torch-pruning performs goes through one function, which is wrapped for the duration of the prune to record the old parameter, the new one, the indices kept and the dimension pruned. No state key is left on a parameter no group holds, which made learn.save() raise inside the wrapped optimizer's state_dict()