HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models

About this repository

This repository provides the JSON files that determine the HICO-DET-SG and V-COCO-SG, the new data splits we created based on the HICO-DET and V-COCO datasets for evaluating the systematic generalization performance of Human-Object Interaction (HOI) detection models.

This repository also provides the source code used to create these JSON files.

The creation process of the new data splits, their statistics, and the evaluation results of four HOI detection models (HOTR, QPIC, FGAHOI, and STIP) on the new splits are available in the following technical note.

Kentaro Takemoto, Moyuru Yamada, Tomotake Sasaki, Hisanao Akima. HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models. arXiv:2305.09948v5, 2024.

JSON files of HICO-DET-SG and V-COCO-SG

The JSON files defining HICO-DET-SG and V-COCO-SG splits are contained in the SGsplits/ folder. You can use them to test the systematic generalization performance of HOI detection models.

How to create JSON files with the source code

1. HOI dataset setup

HICO-DET

You can download the HICO-DET dataset from here.

Instead of using the original annotations files, we use the annotation files provided in this repository. You can download the annotation files from here.

V-COCO

Firstly, clone the repository of V-COCO from here, and then follow the instruction to generate the file instances_vcoco_all_2014.json. Next, download the prior file prior.pickle from here.

Then follow the instruction in here to generate trainval_vcoco.json and test_vcoco.json.

2. Generate SG splits

Firstly, please modify the parameters in main.py.

HICO-DET-SG

Set the random seed as 368, 680 and 750 to generate split 1, 2 and 3, respectively.

Set the ratio as 0.9 for HICO-DET-SG.

Then run:

python main.py

trainval_hico_[output name].json and test_hico_[output name].json will be created in the input directory.

V-COCO-SG

Set the random seed as 564, 966 and 2065 to generate split 1, 2 and 3, respectively.

Set the ratio as 0.7 for V-COCO-SG.

Then run:

python main.py

trainval_vcoco_[output name].json and test_vcoco_[output name].json will be created in the input directory.

Remark

We updated this repository after the presentation at DistShift 2022 (NeurIPS Workshop). The old version used at the time of the workshop can be retrieved from 7c65140, but please use the current version for your own study.

License

• The JSON files determining HICO-DET-SG and V-COCO-SG contained in the SGsplits/ folder are available under the CC0 1.0 License.
• The source code contained in this repository is available under the BSD 3-Clause Clear License.

Citation

If you find the data splits and code helpful for your research, please cite the following.

@misc{Takemoto_2023,
    author    = {Takemoto, Kentaro and Yamada, Moyuru and Sasaki, Tomotake and Akima, Hisanao},
    title     = {{HICO-DET-SG} and {V-COCO-SG}: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models},
    howpublished = {arXiv:2305.09948v5},
    year      = {2024}
}