-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproducing results #41
Comments
Hi Maurits, I just reproduced the results for row 3.15 (Flickr30k ResNet without finetune) using the following command:
The setup is PyTorch 1.4.0 and Python 3.7.1 and I used the changes in the branch pytorch4.1 and python3. The final result as printed is:
Were you running the same command as above? How big is the gap? |
yeah, I use the same command. Do you use a random seed? I use different Python and Torch versions, but that should not give that much of a difference right? I will share my results later today. Thanks again, Maurits |
The seed is not fixed, sorry. Unfortunately, I had not done that in the original code and did not report standard deviations. |
Okay, I've managed to reproduce the results (finally). I still don't know what was the problem in the end. Thanks for your feedback, Maurits Average i2t Recall: 66.9 Average t2i Recall: 55.7 |
Sounds great. Thanks for reporting the result. |
Hi,
First of al, thanks for sharing this great work!
I've difficulties reproducing the results from the paper as a baseline. I will talk about experiment #3.15 in this issue: VSE++ (ResNet), Flickr30k
So what I get from the paper, the config is the following:
My question is: is the image-encoder here trained end-to-end or not. In other words, is ResNet152 only used as a fixed feature extractor, or is it optimized?
According to your documentation, VSE++ (and therefore I assume 3.14) can be reproduced by - only - using the
--max_violation
flag, but I get (way) lower results, do I need the--finetune
flag as well?Thanks,
Maurits
The text was updated successfully, but these errors were encountered: