Progressive End-to-End Object Detection in Crowded Scenes (Deformable-DETR implementation)

Introduction

In this paper, we propose a new query-based detection framework for crowd detection. Previous query-based detectors suffer from two drawbacks: first, multiple predictions will be inferred for a single object, typically in crowded scenes; second, the performance saturates as the depth of the decoding stage increases. Benefiting from the nature of the one-to-one label assignment rule, we propose a progressive predicting method to address the above issues. Specifically, we first select accepted queries prone to generate true positive predictions, then refine the rest noisy queries according to the previously accepted predictions. Experiments show that our method can significantly boost the performance of query-based detectors in crowded scenes. Equipped with our approach, Sparse RCNN achieves 92.0% AP, 41.4% MR^−2 and 83.2% JI on the challenging CrowdHuman dataset, outperforming the box-based method MIP that specifies in handling crowded scenarios. Moreover, the proposed method, robust to crowdedness, can still obtain consistent improvements on moderately and slightly crowded datasets like CityPersons and COCO.

Links

• Iter Sparse R-CNN [repo]
• CVPR 2022 paper [paper]

Models

Experiments of different methods were conducted on CrowdHuman. All approaches take R-50 as the backbone.

Method	#queries	AP	MR	JI
CrowdDet [paper]	--	90.7	41.4	82.4
Sparse RCNN	500	90.7	44.7	81.4
Deformable DETR	1000	91.5	43.7	83.1
Sparse RCNN + Ours [repo]	500	92.0	41.4	83.2
Deformable DETR + Ours (this repo)	1000	92.1	41.5	84.0
Deformable DETR + Swin-L + Ours (this repo)	1000	94.1	37.7	87.1

Installation

The codebases are built on top of Deformable-DETR and Iter-E2EDET.

Steps

1. Install and build libs following Deformable-DETR.

2. Load the CrowdHuman images from here and its annotations from here. Then update the directory path of the CrowdHuman dataset in the config.py.

3. Train Iter Deformable-DETR

bash exps/aps.sh

or for Swin-L backbone:

wget https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth
bash exps/aps_swinl.sh

4. Evaluate Iter Deformable-DETR. You can download the pre-trained model from here for direct evaluation.

# checkpoint path: ./output/model_dump/aps/checkpoint-49.pth
bash exps/aps_test.sh 49
# AP: 0.9216, MR: 0.4140, JI: 0.8389, Recall: 0.9635

or with Swin-L backbone from here:

# checkpoint path: ./output/model_dump/aps_swinl/checkpoint-49.pth
bash exps/aps_swinl_test.sh 49
# AP: 0.9406, MR: 0.3768, JI: 0.8707, Recall: 0.9707

License

Iter Deformable-DETR is released under MIT License.

Citing

If you use our work in your research or wish to refer to the baseline results published here, please use the following BibTeX entries:

@inproceedings{zheng2022progressive,
  title={Progressive End-to-End Object Detection in Crowded Scenes},
  author={Zheng, Anlin and Zhang, Yuang and Zhang, Xiangyu and Qi, Xiaojuan and Sun, Jian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={857--866},
  year={2022}
}