About pretrained detection model

[shine-lcy]

umm, it's me again.
I notice that you release your pretrained faster-rcnn detection model. I download and evaluate it.

CUDA_VISIBLE_DEVICES=1 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/detector_pretest_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/myname/glove MODEL.PRETRAINED_DETECTOR_CKPT ./checkpoints/pretrained_faster_rcnn OUTPUT_DIR ./checkpoints/pretrained_faster_rcnn

btw, I fixed a bug to run above command. The "last_checkpoint" string is the absolute path which contains "kaihua". We need modify some code in "./maskrcnn_benchmark/utils/checkpoint.py".

The result is "Detection evaluation mAp=0.2635", which is far from the value 0.296 mentioned in your paper. Is there something wrong with my evalution command?

[KaihuaTang]

The released pre-trained model is definitely what I used in the paper (cause I deleted other faster-rcnn checkpoints a long time ago). But I forget where I get this 0.296. It could be a mistake like, I reported the mAp on the training set rather than test set. I will check it. Thank you for letting me know.

[KaihuaTang]

I checked my paper where I wrote: "achieved 29.6mAP on VG". I think I might report the mAp on the entire VG dataset (both training set and test set)

[shine-lcy]

I find that the last line of “./checkpoints/pretrained_faster_rcnn/log.txt” is "Detection evaluation mAp=0.2812"(line 3538). I become confused again hhhh

[KaihuaTang]

I find that the last line of “./checkpoints/pretrained_faster_rcnn/log.txt” is "Detection evaluation mAp=0.2812"(line 3538). I become confused again hhhh

That is evaluated on the validation set (5000 imgs). According to my description, I think I reported the mAp on the entire VG. I didn't even remember why, may be I should just report the mAp on test set.

[shine-lcy]

Thanks~

[DiligentPanda]

Hi, actually, I find after training Causal Models (I use motifs) , the detection performance improves to 29.6 because the context is added.

[KaihuaTang]

Hi, actually, I find after training Causal Models (I use motifs) , the detection performance improves to 29.6 because the context is added.

You are right. I think I made a silly mistake. I re-evaluate mAp of pretrained faster-rcnn on VG train/val/test set, and I find the corresponding mAps are: 38.52/26.35/28.14. I will modify the camera-ready version of my paper, reporting mAp 28.14 for the pretrained faster-rcnn on VG test set instead.

[Sonaliam]

"We need modify some code in "./maskrcnn_benchmark/utils/checkpoint.py" "- mentioned by
shine-lcy above.
Hi,
Can anyone please tell me, which modification is needed in above code to run testing code?
I am getting test results almost zeros.
I modified absolute path in last_checkpoint file. Can anyone share modified following file?
"./maskrcnn_benchmark/utils/checkpoint.py"