-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The evaluation result is not good. #2
Comments
Can you tell me your device setup? Also, please try our checkpoint from this URL. |
pytorch: 1.8 |
I think it's pretty much not the device problem, but I also didn't have much experience with RTX3090. Also, I personally experienced poor results occasionally but it wasn't that often, so you can try another experiment with same settings. Please reopen this issue if you still have problem. |
Hi, thank you for having more question about our work.
After I've got your email I run on my own machine with TITAN RTX (24GB) and got these results.
Expr - UACANet-L 100%|██████████████████████████████████████████████████| 5/5 [49:08<00:00, 589.76s/it]
dataset meanDic meanIoU wFm Sm meanEm mae maxEm maxDic maxIoU meanSen maxSen meanSpe maxSpe
----------------- --------- --------- ----- ----- -------- ----- ------- -------- -------- --------- -------- --------- --------
CVC-300 0.902 0.835 0.884 0.933 0.972 0.006 0.975 0.906 0.838 0.962 1.000 0.992 0.995
CVC-ClinicDB 0.932 0.884 0.931 0.945 0.979 0.007 0.982 0.935 0.888 0.946 1.000 0.992 0.996
Kvasir 0.905 0.851 0.896 0.915 0.947 0.025 0.950 0.907 0.853 0.909 1.000 0.985 0.989
CVC-ColonDB 0.762 0.690 0.755 0.842 0.874 0.034 0.876 0.765 0.692 0.771 1.000 0.931 0.934
ETIS-LaribPolypDB 0.714 0.645 0.691 0.829 0.845 0.016 0.847 0.717 0.647 0.760 1.000 0.905 0.909
Results shows some difference between the results from my paper but overall, plausible while yours aren’t.
Here’s what I thought. Polyp Segmentation task has small number images to train according to PraNet, the original work of mine. So, it varies the results quite a lot. I’d personally did almost 10 different experiments with the same setting to obtain best results from the paper.
However, since you’d run the same setting of mine twice and got similar results which are seems to be quite different than mine when it comes to the paper results or my very recent experiment shown above, I think it’s the number of batch sizes then. I would recommend you to change the number of batch size into at least 16. I will also do some more experiments to make sure that the problem comes from batch size and let you know the results.
|
Hi, I had two additional experiments with batchsize 16 and 8 and here are the results. I never knew that small batchsize would affect ETIS dataset this much, but turns out it does. |
Thank you a lot for the help. I just dive into this area for a while. Due
to the limitation of my device, I can only run the code if the batch size
is 8.
When the batch size is changed to 16, CUDA out of memory. Setting the
batch size to 8, I will try to run the code again. I want to check it again
with the same condition.
Again, thanks for your help.
Taehun Kim ***@***.***> 于2021年9月24日周五 上午10:08写道:
… Reopened #2 <#2>.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARPRTJAKVCMQED2ESZXO2VLUDPMRJANCNFSM5B6TV6XA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
Hi, |
Following are the results: CVC-300 0.909 0.846 0.895 0.937 0.977 0.005 0.980 0.913 0.850 0.960 1.000 0.992 0.996 dataset meanDic meanIoU wFm Sm meanEm mae maxEm maxDic maxIoU meanSen maxSen meanSpe maxSpe CVC-300 0.909 0.846 0.895 0.937 0.976 0.005 0.979 0.912 0.849 0.958 1.000 0.992 0.996 These are still not as claimed on paper (however, there is an improvement with batch size 32). |
I'm closing this issue since there are no other updates. |
I run the commad:
However, the result listed as follows:
The official result of UACANet-L mentioned at README is
Why is the result I run so bad? I didn't change any configuration file.
The text was updated successfully, but these errors were encountered: