Zero-Shot Out-of-Vocabulary Object Detection

[2601.22685] OOVDet: Low-Density Prior Learning for Zero-Shot Out-of-Vocabulary Object Detection (arxiv.org)

• Installation

git clone https://github.com/binyisu/OOVDet.git
cd OOVDet/

conda create -n oovdet python=3.8 -y
conda activate oovdet

pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113

python -m pip install -e detectron2-0.3

pip install -r requirements.txt

• Prepare datasets

You should download：

• train and val set of COCO2017

• trainval and test set of VOC2007、VOC2012

following the structure described below：

datasets/
  coco/
  VOC20{07,12}/

In coco：

coco/
  annotations/
    instances_{train,val}2017.json
    person_keypoints_{train,val}2017.json
  {train,val}2017/

In VOC20{07,12}：

VOC20{07,12}/
  Annotations/
  ImageSets/
    Main/
      trainval.txt
      test.txt
  JPEGImages/

Then we generate all datasets for ZOOV:

bash prepare_oov_voc_coco.sh

• Prepare models

Follow ".\offline_rpn_weights\README.md" and ".\pretrained_ckpt\regionclip\README.md" to prepare pretrained models.

• Prepare concepts

Please download the required file from Google Drive and place it in:

./concepts/

• Running

  • OOV-COCO dataset settings:

    bash OOV-COCO.sh

  • OOV-VOC dataset settings:

    bash OOV-VOC.sh

Inference / Testing with app.py

You can directly test the trained model using:

python app.py

Before running, please download the trained model weights (model_final.pth) from Google Drive and place them in:

./output/voc/rn50x4/

Note that the comm.py, rpn.py, proposal_utils.py and batch_norm.py are modified version based on the Release v0.3 · facebookresearch/detectron2 (github.com)

All our experiments were conducted on a single NVIDIA 1080Ti, with a batch size of 1 for base class training and a batch size of 1 for novel class fine-tuning.