Question on evaluation result

[lizhyuxi]

I performe the given Inference code but get the different evaluation result from that in the paper:
My gIoU is 66.3407, cIoU is 63.0991, but in the paper they are respectively 63.60 and 62.42

Here follow my running code, file directory and the output. Is there anything wrong? Thank you.

!python train_net.py
--config-file configs/referring_swin_base.yaml
--num-gpus 1 --dist-url auto --eval-only
MODEL.WEIGHTS "/content/ReLA/gres_swin_base.pth"
OUTPUT_DIR "/content/ReLA"

[yahooo-m]

The same results！

[henghuiding]

Hi, thank you for raising this issue.

The results are correct. It is expected that the performance of the released models is slightly better than that reported in our CVPR paper. We have optimized the model after the submission of the camera-ready version, for example here. For the original version please kindly refer to this branch.

[qiulesun]

@henghuiding

When I reproduce your results using gres_r50.pth model, I encounter the following problem. Could you provide some suggestions to fix it ?

inference cmd：
python train_net.py
--config-file configs/referring_R50.yaml
--num-gpus 2 --dist-url auto --eval-only
MODEL.WEIGHTS modes/gres_r50.pth
OUTPUT_DIR modes/gres_r50-eval

log.txt

[changliu19]

Hi, This issue is likely to be caused by running out of CPU RAM. Try closing other running programs and retry. Best, Liu

…

________________________________ From: Qiule Sun ***@***.***> Sent: Thursday, July 27, 2023 10:03:04 PM To: henghuiding/ReLA ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [henghuiding/ReLA] Question on evaluation result (Issue #10) @henghuiding<https://github.com/henghuiding> When I reproduce your results using gres_r50.pth model, I encounter the following problem. Could you provide some suggestions to fix it ? inference cmd： python train_net.py --config-file configs/referring_R50.yaml --num-gpus 2 --dist-url auto --eval-only MODEL.WEIGHTS modes/gres_r50.pth OUTPUT_DIR modes/gres_r50-eval [image]<https://user-images.githubusercontent.com/17628841/256561193-71978526-453e-4c91-886c-6728d5619ebb.png> — Reply to this email directly, view it on GitHub<#10 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AM7BARLTLERFTH45YBXOVYTXSJYJRANCNFSM6AAAAAA2GCSB4M>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[qiulesun]

@ntuLC @henghuiding

My CPU RAM is 64GB and I already close all programs when running released code.
But whether I fine-tune model or use released model to reproduce the results, there's always this problem. I have no idea to fix it. Can you give me some suggestions ?

[changliu19]

@ntuLC @henghuiding

My CPU RAM is 64GB and I already close all programs when running released code. But whether I fine-tune model or use released model to reproduce the results, there's always this problem. I have no idea to fix it. Can you give me some suggestions ?

Hi @qiulesun ,

You may monitor the system memory usage using top whilst running, or check your system log to confirm whether it is an OOM issue.

[lcl-git-3d]

Hello! Why my evaluation result is not same with you? @qiulesun