-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bad performance on NLVR2 #1
Comments
Many thanks for running the experiment and point this issue out!!! I am now running a verified experiment and would let you the result tomorrow morning. PyTorch VersionMy initial guess is the PyTorch version. Could you help to try torch==1.0.1? Installation command:
I found that I used an old virtualenv with an old PyTorch version. I was supposing that PyTorch should be backward-compatible in computing gradients but it seems not the case. By the way, here is a full list of my virtualenv. I believe that the only difference might be the torch version.
If so, it's really strange but I will update Raw FeatureAnd could you also try to use the raw feature from our server in replace of the feature from zip files with the command:
In case there are some broken zip files. |
Thanks for the update! Yes, it does look like the torch version was the issue. (So far I have re-trained for 1 epoch using pytorch 1.0.1 and got a validation accuracy of 67.86.) Previously, I was using version 1.1.0. |
Hi, thanks for releasing your code! I'm not able to reproduce your fine-tuning result on NLVR2. I followed your instructions by downloading the pre-trained model, downloading the image features, pre-processing the nlvr2 JSON files, and running the nlvr2_finetune.bash script as is. However, I get the following results, which are much lower than the result you reported. Do you know why this might be happening?
Epoch 0: Train 52.32
Epoch 0: Valid 50.86
Epoch 0: Best 50.86
Epoch 1: Train 50.50
Epoch 1: Valid 49.14
Epoch 1: Best 50.86
Epoch 2: Train 50.56
Epoch 2: Valid 49.31
Epoch 2: Best 50.86
Epoch 3: Train 54.83
Epoch 3: Valid 51.65
Epoch 3: Best 51.65
The text was updated successfully, but these errors were encountered: