Tiny BERT saving issue

[percevalw]

Description

Running the training tests with pretrained BERT "hf-internal-testing/tiny-bert" causes this issue:

You are trying to save a non contiguous tensor: `encoder.layer.0.attention.self.query.weight` 
which is not allowed. It either means you are trying to save tensors which are reference of each 
other in which case it's recommended to save only the full tensors, and reslice at load time, or 
simply call `.contiguous()