Getting error during training

[arikhalperin]

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Which means that the data did not move to GPU.

My code:

torch.device("cuda")

model = SpeechRecognitionModel("jonatasgrosman/wav2vec2-large-xlsr-53-spanish", device="cuda")
processor_ref = Wav2Vec2Processor.from_pretrained("jonatasgrosman/wav2vec2-large-xlsr-53-spanish")
token_list = list(processor_ref.tokenizer.encoder.keys())
token_set = TokenSet(token_list)

train_set = []
eval_set = []

train_set, eval_set = add_sealed_data_set(train_set, eval_set, config[environment][SAMPLES_DIR])

training_arguments = TrainingArguments()
training_arguments.overwrite_output_dir = True
training_arguments.per_device_train_batch_size = 128
training_arguments.per_device_eval_batch_size = 128

model.finetune(
    config[environment][MODEL_OUTPUT_DIR],
    train_data=train_set,
    eval_data=eval_set,  # the eval_data is optional
    token_set=token_set,
    training_args=training_arguments
)

Managing to work around this by adding a move to cuda of my dataset inside huggingsound code. If I can make it work I'll create a PR

[jonatasgrosman]

Hi @arikhalperin, Did you manage to solve this issue? I couldn't reproduce your error on my machine. Maybe this issue is related to your environment. Please send me more info about the version of your Cuda, PyTorch, etc. so that may I can help you to figure out what's going on :)

Another good option is to try to reproduce this issue on a Colab and send me the link

[arikhalperin]

Sorry about the delay. It updated CUDA and pytorch to match and it was resolved.