Cannot Finetune the model more than once

[VargEM]

Hi there, there is a problem I faced when dealing with this model. I loaded your wav2vec2 model and finetuned it on another small dataset (10 short voices), everything was okay with the training (although the prediction of the new trained model dropped significantly), but when I wanted to train the new trained model for the second time on another small dataset, I faced this error sometimes:

size mismatch for lm_head.weight: copying a param with shape torch.Size([40, 1024]) from checkpoint, the shape in current model is torch.Size([38, 1024]). size mismatch for lm_head.bias: copying a param with shape torch.Size([40]) from checkpoint, the shape in current model is torch.Size([38]).

despite the very small amount of data and small batch size and all other stuff, sometimes I face this error too:

RuntimeError: CUDA out of memory. Tried to allocate 94.00 MiB (GPU 0; 14.76 GiB total capacity; 12.95 GiB already allocated; 91.75 MiB free; 13.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is there any way out? Thanks a lot.