Accepted at WACV 2023 link Previous Version:
Disclaimer: This repository is for research purposes only. It is designed to maintain reproducibility of the experiments described in "The Box Size Confidence Bias Harms Your Object Detector".
Download COCO2017 annotations for train, val, and test-dev from here
and move them into the folder structure like this (alternatively change the config in config/all/paths/annotations/coco_2017.yaml
to your local folder structure):
.
└── data
└── coco
└── annotations
├── instances_train2017.json
├── instances_val2017.json
└── image_info_test-dev2017.json
Generate Detections (or Download them here)
Generate detections on the train, val, and test-dev COCO2017 set, save them in the COCO file format as JSON files.
Move detections to data/detections/MODEL_NAME
, see config/all/detections/default_all.yaml
for all the used detectors and to add other detectors.
The official implementations for the used detectors are:
- CenterNet(HG) [Link]
- Faster-RCNN, YOLOv3-320, DETR, RetinaNet, CenterNet (RN-18), etc. [Link]
- SSD [Link]
- CenterNet2 [Link]
- YOLOv5x [Link]
- EfficientDet D0-7 [Link]
To generate the Detections for CenterNet with Hourglass backbone first follow the installation instructions. Then download ctdet_coco_hg.pth
to /models
from the official source
Then generate the detections from the /src
folder:
# On val
python3 test.py ctdet --arch hourglass --exp_id Centernet_HG_val --dataset coco --load_model ../models/ctdet_coco_hg.pth
# On test-dev
python3 test.py ctdet --arch hourglass --exp_id Centernet_HG_test-dev --dataset coco --load_model ../models/ctdet_coco_hg.pth --trainval
# On train
sed '56s/.*/ split = "train"/' test.py > test_train.py
python3 test_train.py ctdet --arch hourglass --exp_id Centernet_HG_train --dataset coco --load_model ../models/ctdet_coco_hg.pth
The scaling for TTA is set via the "--test_scales LIST_SCALES"
flag. So to generate only the 0.5x-scales: --test_scales 0.5
To generate the de detection files using mmdet, first follow the installation instructions. Then download specific model weights, in this example retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth
to PATH_TO_DOWNLOADED_WEIGHTS
and execute the following commands:
python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/train2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/train2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/instances_train2017.json'
python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/val2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/val2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/instances_val2017.json'
python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/test-dev2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/test2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/image_info_test-dev2017.json'
pip3 install -r requirements.txt
# Faster coco evaluation (used if available)
pip3 install fast_coco_eval
# Parallel multi-runs, if enough RAM is available (add "hydra/launcher=joblib" to every command with -m flag)
pip install hydra-joblib-launcher
Most of the experiments are performed using the CenterNet(HG) detections to change the detector add detections=OTHER_DETECTOR
, with the location of OTHER_DETECTORs detections specified in config/all/detections/default_all.yaml
.
The results of each experiment are saved to outputs/EXPERIMENT/DATE
and multirun/EXPERIMENT/DATE
in the case of a multirun (-m flag).
# original histogram binning calibration curve
python3 create_plots.py -cn plot_org_hist_bin
# modified histogram binning calibration curve:
python3 create_plots.py -cn plot_mod_hist_bin
python3 calibrate.py -cn ablate_modified_hist
python3 calibrate.py -cn ablate_metrics "seed=range(4,14)" -m
Plot of calibration curve:
# on validation data
python3 create_plots.py -cn plot_miscal name="plot_miscal_val" split="val"
# on train data:
python3 create_plots.py -cn plot_miscal name="plot_miscal_train" split="train" calib.conf_bins=20
python3 calibrate.py -cn explore_train
-
Generate detections (on train and val split) for each scale-factor individually
(CenterNet_HG_TTA_050, CenterNet_HG_TTA_075, CenterNet_HG_TTA_100, CenterNet_HG_TTA_125, CenterNet_HG_TTA_150)
and for complete TTA(CenterNet_HG_TTA_ens)
-
Generate individually calibrated detections..
python3 calibrate.py -cn calibrate_train name="calibrate_train_tta" detector="CenterNet_HG_TTA_050","CenterNet_HG_TTA_075","CenterNet_HG_TTA_100","CenterNet_HG_TTA_125","CenterNet_HG_TTA_150","CenterNet_HG_TTA_ens" -m
-
Copy calibrated detections from
multirun/calibrate_train_tta/DATE/MODEL_NAME/quantile_spline_ontrain_opt_tradeoff_full/val/MODEL_NAME.json
todata/calibrated/MODEL_NAME/val/results.json
forMODEL_NAME
in(CenterNet_HG_TTA_050, CenterNet_HG_TTA_075, CenterNet_HG_TTA_100, CenterNet_HG_TTA_125, CenterNet_HG_TTA_150)
. -
Generate TTA of calibrated detections
python3 enseble.py -cn enseble
python3 calibrate.py -cn calibrate_train name="ablate_iou" "iou_threshold=range(0.5,0.96,0.05)" -m
python3 calibrate.py -cn calibrate_train name="calibrate_all_models" detector=LIST_ALL_MODELS -m
The test-dev predictions are found in multirun/calibrate_all_models/DATE/MODEL_NAME/quantile_spline_ontrain_opt_tradeoff_full/test/MODEL_NAME.json
and can be evaluated using the official evaluation sever.
python3 calibrate.py -cn ablate_metrics_extended "seed=range(4,14)" -m
# Results for B0, C0
python3 calibrate.py -cn calibrate_train
# Results for B0, C1
python3 calibrate.py -cn calibrate_train_larger_cbins
# Results for B0 union B1, C0
python3 calibrate.py -cn calibrate_train_larger_bbins
# Results for B0 union B1, C0 union C1
python3 calibrate.py -cn calibrate_train_larger_cbbins
python3 calibrate.py -cn calibrate_train name="influence_modelsize" detector="Efficientdet_D0","Efficientdet_D1","Efficientdet_D2","Efficientdet_D3","Efficientdet_D4","Efficientdet_D5","Efficientdet_D6","Efficientdet_D7" -m
@inproceedings{gilg2023box,
title={The Box Size Confidence Bias Harms Your Object Detector},
author={Gilg, Johannes and Teepe, Torben and Herzog, Fabian and Rigoll, Gerhard},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={1471--1480},
year={2023}
}