-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The test result from uploaded FSC147.pth does not match the result of the paper #23
Comments
I found the reason. FSC_test_cross(few_shot).py |
Hi @Jyerim, sorry for introducing this mistake. Can you share your new results? My experiments with the original test code, without this error, are still quite different with respect to the results reported in the paper, and I'd like to know if it is due to the different PyTorch/timm version. I tested the few shot counting task with the uploaded weights
You can check the difference between the two scripts here. |
I don't think the difference of PyTorch/timm version will make the result vary so much. Maybe there is something wrong with the 0-shot and 3-shot setting? Or maybe you din't use test-time normalisation? In Issue #7, he used the checkpoint I uploaded and got the result of MAE: 12.44, RMSE: 89.86, so there might be something wrong with the test script you used. |
Thank you @Verg-Avesta, I managed to obtain the results you mentioned. |
The test result from uploaded FSC147.pth does not match the result of the paper.
The inference result for FSC147 test data is MAE 15.71, RMSE 104.99. I use the FSC147 fine-tuned weights which you upload on the document.
However, the zero shot result is similar to the result of the paper.
Few shot : 15.71 / 104.99
Zero shot: 14.70 / 106.87
I ran the evaluation code with torch 1.10, timm (0.3.2, 0.4.5). I tried 0.3.2 and 0.4.5 version of timm.
Is there any way to fix this issue?
The text was updated successfully, but these errors were encountered: