Add parametrization for normalization

[y-prudent]

Prerequisite: Torch parametrization tutorial

Features

• Rewrite bjorck, frobenius, and lconv normalizations so that they use torch parametrization instead of forward pre hooks.
• Faster inference in the model.eval() mode (when model.training is set to False, spectral and bjorck normalizations use cached tensors to perform free normalizations).
• Vanilla model conversion can be done on any model (not only Sequential) with vanilla_model. Be careful, this is an in-place conversion!
• Torch parametrize.cached() feature is now also usable on Lipschitz layers, allowing to save memory and compute when the same kernel is applied multiple times in an inference step (very useful for RNNs, multi-level convolutions, etc.). Here is how to use it:

import torch.nn.utils.parametrize as P

with P.cached():
    y1 = lip_layer(x1)    # at the first call, compute normalization and save the reparametrized weights
    y2 = lip_layer(x2)    # at the second call, reuse saved reparametrized weights

⚠️ Important note:

Models using parametrized modules can only be serialized through state_dict(). So torch.save(model, PATH) is not possible anymore and will raise an error. Instead, save and load your models like this:

# save model
torch.save(
    {
        'model_state_dict': model.state_dict(),
        'model_kwargs': config,  # arguments used to build the model
    }, PATH
)

# load model
checkpoint = torch.load(PATH)
model = TheModelClass(**checkpoint["model_kwargs"])
model.load_state_dict(checkpoint['model_state_dict'])

For more information, check this torch tutorial.

TODO

• Rewrite bjorck, frobenius and lconv normalizations with parametrization
• Manage the suppression of individual parametrizations (torch native functions only allow the deletion of all the parametrizations at once)
• Manage the parametrization of lconv_norm that depends on the input shape (=> solution: forward pre hooks + parametrization)
• Also parametrize the global lipschitz coef multiplication
• Write associated tests
• Remove deprecated hook relative files
• Check that vanilla_export works well
• Check parametrize.cached() feature

[cofri]

Very interesting PR!
Just a small suggestion for vanilla_export import

[cofri]

From the statuses of Python versions, Python 3.7 is already deprecated. Maybe we can use a more powerful test matrix with more recent Python versions (and also PyTorch versions?).
It's a more important change and it could be postponed to a future PR