This repository contains the code used to generate the results reported in the paper: Concept-Conditioned-Object-Detector
Some of our code is based on DynamicHead. Thanks!
@article{rigoni2024object,
title={Object search by a concept-conditioned object detector},
author={Rigoni, Davide and Serafini, Luciano and Sperduti, Alessandro},
journal={Neural Computing and Applications},
pages={1--21},
year={2024},
publisher={Springer}
}
This project uses the conda
environment.
conda create -n ccobj pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.1 torchtext=0.11.0 -c pytorch -c conda-forge
conda activate ccobj
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html
python -m pip install -e ./
pip install nltk
pip install numpy
pip install setuptools==59.5.0 # https://stackoverflow.com/questions/70520120/attributeerror-module-setuptools-distutils-has-no-attribute-version
pip install wandb
The project is structured as follows:
datasets
: contains datasets;concept
: contains the mapping from the object detector categories to the WordNet concepts and the concepts vocabulary;configs
: contains the configuration files to reproduce each experiment presented in the paper;extra
: contains our main code;results
: contains the results;pretrained
: contains pre-trained checkpoints;tools
: contains auxiliary files.
For more details on the datasets used and how to download the online hosted versions, refer to this guide: README
As stated in the paper, we adopted pre-trained embeddings for each WordNet concepts using th code of this repository: WordNet_Embeddings.
The pre-trained weights must be placed in the ./concepts/
folder. Download them using the following link: Download.
The final concept
folder structure should be the following:
|-- concept
|-- coco_to_synset.json
|-- vg_to_synset.json
|-- oid_to_synset.json
|-- vocab.json
|-- wn30_holE_500_150_0.1_0.2_embeddings.pickle
Use the following command for training:
python train_net.py --config {TRAINING_CONFIG_FILE}
If necessary, specify the number of machines and GPUs to use:
python train_net.py --config {TRAINING_CONFIG_FILE} --num-gpus 4 --num-machines 4 --dist-url tcp://login1234:19206
NOTE: the TRAINING_CONFIG_FILE
may need to be edited to correctly load pre-trained models and the training dataset.
Two functions can be used to evaluate models. The former performs model evaluation without applying the post-processing filtering before evaluation:
python train_net.py --config {TEST_CONFIG_FILE} --eval-only EVALUATOR_TYPE 'default'
The second performs post-processing filtering of the bounding boxes that the model predicts. In practice, the COCO evaluator loads all the bounding boxes predicted by the model, filters them, and then continues with the COCO evaluation.
python train_net.py --config {TEST_CONFIG_FILE} --eval-only EVALUATOR_TYPE 'postProcessing'
NOTE: Configuration files (i.e, TEST_CONFIG_FILE
), for testing presents a test set for evaluation that is not always correct. For this reason, during evaluation indicate the dataset to adopt:
python train_net.py --config {TEST_CONFIG_FILE} --eval-only EVALUATOR_TYPE 'default' DATASETS.TEST '("coco_2017_val_subset_old",)'
All the datasets can be seen in the file: ./extra/datasets.py'
.
Unfortunately, there is not enough space available to host all checkpoints. However, we provide the models with the best performance.
Config | Model | Backbone | Dataset | Weight |
---|---|---|---|---|
cfg | DynamicHead | Swin-Tiny | COCO | weight |
cfg | Concept DynamicHead | Swin-Tiny | COCO | weight |
cfg | DynamicHead | Swin-Tiny | VG | weight |
cfg | Concept DynamicHead | Swin-Tiny | VG | weight |
For any questions and comments, contact davide.rigoni.2@phd.unipd.it.
MIT