Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting very low mIoU using pre-trained weights for DMN #21

Closed
tusharkumar91 opened this issue Dec 31, 2018 · 6 comments
Closed

Getting very low mIoU using pre-trained weights for DMN #21

tusharkumar91 opened this issue Dec 31, 2018 · 6 comments

Comments

@tusharkumar91
Copy link

Hi,

I am attempting to use the DMN model with the pre-trained weights provided. However, whenever I evaluate the model using the below-mentioned command for UNC dataset using the pre-trained weights downloaded from the link provided in the ReadMe, I get a very low mIoU(close to 0.11).

Pre-trained weights( for unc dataset)

Command Used for evaluating the model

python -u -m dmn_pytorch.train --data referit_data --dataset unc --val testA --backend dpn92 --num-filters 10 --lang-layers 3 --mix-we --snapshot ./snapshots/dmn_unc_weights.pth --epochs 0 --eval-first --high-res

Result of Evaluation(using above command)

Evaluation done. Elapsed time: 343.907 (s) 
Maximum IoU: 0.1159133240581 - Threshold: 0.0000000000000

I have tried with other datasets also but I am unable to get anywhere close to the performance numbers mentioned in the ReadMe using the pre-trained weights. I am not sure what the issue is here. Below I provide the different versions of packages/libraries I am using.

Dependency Versions

Python 3.7
Pytorch 1.0
SRU 2.1.2 (latest version)
Cuda 9.0

Please note that I am getting low numbers for all datasets and have only provided UNC dataset(and numbers) as an example.

@tusharkumar91
Copy link
Author

Thanks to the help from @andfoy, I am now able to get the same mIoU numbers as mentioned in Readme using the pre-trained weights. The issue was that I was using the latest version of SRU. On suggestion from @andfoy, I switched to the version of SRU as of this commit. Once I switched to this version, running the below mentioned command gave me the following results.

Command Used for evaluating the model

python -u -m dmn_pytorch.train --data referit_data --dataset unc --val testA --backend dpn92 --num-filters 10 --lang-layers 3 --mix-we --snapshot ./snapshots/dmn_unc_weights.pth --epochs 0 --eval-first --high-res

Result of Evaluation(using above command)

UNC testA 
Maximum IoU: 0.5212885141373 - Threshold: 0.2250000000000

UNC testB
Maximum IoU: 0.4338888525963 - Threshold: 0.1750000000000


UNC+ testA
Maximum IoU: 0.4425530731678 - Threshold: 0.3750000000000

UNC+ testB
Maximum IoU: 0.3249503076077 - Threshold: 0.2750000000000

Referit test
Maximum IoU: 0.5280885696411 - Threshold: 0.4250000000000

Gref val
Maximum IoU: 0.3764866888523 - Threshold: 0.2000000000000

As one can see that the above numbers are similar to that provided in the readme file. So using the version of SRU provided in the ReadMe file instead of using the latest version fixed the issue that I was facing and enables me to reproduce the results mentioned in ReadMe.

tusharkumar91 pushed a commit to tusharkumar91/Simple-Recurrent-Units that referenced this issue Jan 3, 2019
…of DMN.

BCV-Uniandes/DMS#21
Hence switching to the latest compatiable verison of SRU.
However the issue with older version of SRU is that it becomes
incompatiable with latest version of pynvtrc.
The new version of pynvrtc assumes bytes objects instead of str objects as input to Program(),
hence removing the encode calls in cuda_functional to have this older version
of SRU also be compatiable with latest version of pynvtrc
@13331112522
Copy link

@andfoy Great thanks for your timely reply. I followed the instruction to update the SRU to 2.1.3 which already support Prnvrtc 9.2. But my problem is still there.

python -u -m dmn_pytorch.train_gai --data referit_data --dataset referit --val val --backend dpn92 --num-filters 10 --lang-layers 3 --mix-we --save-folder weights --snapshot dmn_referit_weights.pth --epochs 0 --eval-first

Evaluation done. Elapsed time: 1035.489 (s)
Maximum IoU: 0.1528985351324 - Threshold: 0.0000000000000

I couldn't add --high-res because of out of memory problem, I had nvidia 2080 with 8G memory. I don't know the reason why it's so easy to leak.
Thanks for your consideration.

@13331112522
Copy link

P.S: Dependency:
Python 3.6,8
Pytorch 1.0.1
CUDA 10
sru 2.1.3
prnvrtc 9.2
Window 10

@andfoy
Copy link
Contributor

andfoy commented May 16, 2019

I followed the instruction to update the SRU to 2.1.3 which already support Prnvrtc 9.2

Actually, you need to install the SRU version from this commit:

pip install -U git+https://github.com/taolei87/sru.git@43c85ed --no-deps

I couldn't add --high-res because of out of memory problem, I had nvidia 2080 with 8G memory.

Try reducing the image size of your inputs, using --size (it defaults to 512 pixels on the largest side). Using the high-res pretrained weights on a low-res fashion won't work, as those weights use the Upsampling module, whereas low-res mode does not.

@13331112522
Copy link

13331112522 commented May 16, 2019

Followed your instruction and reinstall sru-0.0.1, remove encode() with 348 line.

python -u -m dmn_pytorch.train_gai --data referit_data --dataset referit --val val --backend dpn92 --num-filters 10 --lang-layers 3 --mix-we --save-folder weights --snapshot dmn_referit_weights.pth --epochs 0 --eval-first --high-res

Still mIoU around 0.15. @andfoy

@13331112522
Copy link

I resize from 512 to 384, it works well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants