Invalid tensor shape

[peregilk]

I keep getting the following error at the end of the first epoch:
"RuntimeError: Input tensor at index 1 has invalid shape [22, 44], but expected [22, 46]". This happens on a custom dataset. However, the dataset is thoroughly cleaned and should be valid.

The error happens in: comm.py, line 231

Any idea what might be causing this?

[peregilk]

Deleting the last line in the dataset actually fixed the error in my case.

[epiphany711]

Do you know the reason of this problem? I met the similar mistake, and I don't know how to fix it. The dataset I used was wiki1m_for_simcse.txt

[peregilk]

no. I did not dig into this. I started debugging this, and just wanted to see if the shape of the vector changed if the dataset size changed. Would give me a hint about what was wrong. Then it just solved the issue. Seems like the last batch is filled incorrectly. Most likely trivial. I am new to SimCSE. Someone knowing the code could fix this easier.

[haoliutj]

I have the same issue, deleting the last line does not fix my error. Wondering if anyone has better solutions?

"comm.py", line 235, in gather
return torch._C._gather(tensors, dim, destination)
RuntimeError: Input tensor at index 3 has invalid shape [32, 32], but expected [32, 34]"

[Dicer-Zz]

I meet the same issue for my custom dataset. Can anyone give some suggestion？

[Dicer-Zz]

I found the issue has been solved in #148 . In my case, using only single GPU is working in the unsupervised setting.

[haoliutj]

I solved by running the scripts with multiple GPU. the issue is that I was running the script for single gpu on 4 gpu cards that cause the shape errors. I

[gaotianyu1350]

Thanks @haoliutj @Dicer-Zz for answering this! This is likely caused by running the single gpu script on multiple GPUs. Please use the corresponding script for different number of GPUs.