-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some questions for the paper #11
Comments
See #6. Thanks! |
@zdou0830 Thanks! I have another question about the codes. I tried your pretrained model to finetune VQA v2 with default setting. It should be more than 80.. Could you share your experiment settings of finetuning VQA v2 and pretraining tasks? |
The command should be
Hope this thread would help #7. |
This setting is the same as mine ... |
If I use your finetuned model to do the test only task of VQA v2, the result is 77.66. |
You can try testing the last checkpoint and submitting the resulting json file to evalai. |
The result from evalai is 71.53. |
The VQA dataset can be downloaded here: https://drive.google.com/file/d/1qT7YWHpLg-fAL43daKlOsYx2EbbQk--d/view?usp=sharing. The training command is
The testing command is
The provided VQA-finetuned checkpoint is trained in this way, so if you follow these steps correctly, you should be able to get a score of ~77.6 on test-dev. I didn't look at the dev scores and the number of training epochs was set to 10 as in config.py. The fine-tuning took about 2 days on 8 V100s for reference. |
Ok, I will have a try! Thank you for your patient. |
I found a problem. That might be the reason. May you share your pretraining log? |
I didn't save the logs, but I did pre-train the models with 1/2/4 nodes and there were no significant differences, so I'd suggest you to debug your multi-node training settings. |
I also met the same problem, which probably results from multi-node training settings. I use slurm to multi-node train. May you share your train bash file, if you also use slurm. |
I didn't use slurm, but I uploaded the running file for distributed training on Microsoft machines (https://github.com/zdou0830/METER/blob/main/azure_distributed_run.py). Not sure if this is helpful. |
ok, thank u~ |
@jiyt17 Hello, have you solved the problems you encountered before? |
@wanng-ide Hello, have you solved the problems you encountered before? |
What is the difference between the score in Table 5 and Table 8?
77.19 in Table 5 results on test-dev set of VQAv2, and,
77.68 in Table 8 results on test-dev set of VQAv2.
The text was updated successfully, but these errors were encountered: