Running HICO Evaluation

[vjsrinivas]

Hi @fredzzhang, thank you for your great work!
I am trying to reproduce the reported mAPs of the pretrained COCO version, but I'm confused on how to get the HICO evaluation running.

I ran the network through test2015 successfully and generated both JSON and .mat files. I tried running the mat files with eval_run.m from HO-RCNN but got very lower mAPs:

Is there something I am missing? I would appreciate any help.

[fredzzhang]

Hi, @vjsrinivas

I did not run the test in the known object (ko) setting. But from the default setting (def), the performance 0.608 looks like it's using a randomly initialised model, instead of our trained model.

So just to make sure, have you downloaded the checkpoint of our trained model? And when generating the .mat files, have you correctly specified the path to the checkpoint?

1. Download the checkpoint

cd /path/to/spatio-attentive-graphs/download
bash download_checkpoint.sh

The checkpoint should have been downloaded to /path/to/spatio-attentive-graphs/checkpoints/spatio-attentive-graph-hicodet-e11.pt

2. Run the caching script

cd /path/to/spatio-attentive-graphs
CUDA_VISIBLE_DEVICES=0 python cache.py --model-path checkpoints/spatio-attentive-graph-hicodet-e11.pt

When running this script, you should see a printed message that says Loading model from checkpoints/spatio-attentive-graphs-hicodet-e11.pt

Let me know if this resolves the issue.

Cheers,
Fred

[vjsrinivas]

Ahh, I did not check if the file path was correct! It should be checkpoints/spatio-attentive-graphs-hicodet-e11.pt.
I ran it with the right name and was able to match the reported stats:

Also, is there a reason why the program doesn't just error out if the weight path doesn't exist?

[fredzzhang]

Glad it resolved your issue. When the given path is empty or non-existent, a randomly initialised model will be used. I guess I could add some warnings when the given path is not empty but incorrect. Will do that later.

Cheers,
Fred