Object Detectors in the Open Environment: Challenges, Solutions, and Outlook

This is the repository of Object Detectors in the Open Environment: Challenges, Solutions, and Outlook. For details, please refer to: [Paper]

Feel free to contact us or pull requests if you find any related papers that are not included here.

Abstract

With the emergence of foundation models, deep learning-based object detectors have shown practical usability in closed set scenarios. However, for real-world tasks, object detectors often operate in open environments, where crucial factors (\eg, data distribution, objective) that influence model learning are often changing. The dynamic and intricate nature of the open environment poses novel and formidable challenges to object detectors. Unfortunately, current research on object detectors in open environments lacks a comprehensive analysis of their distinctive characteristics, challenges, and corresponding solutions, which hinders their secure deployment in critical real-world scenarios. This paper aims to bridge this gap by conducting a comprehensive review and analysis of object detectors in open environments. We initially identified limitations of key structural components within the existing detection pipeline and propose the open environment object detector challenge framework that includes four quadrants (\ie, out-of-domain, out-of-category, robust learning, and incremental learning) based on the dimensions of the data / target changes. For each quadrant of challenges in the proposed framework, we present a detailed description and systematic analysis of the overarching goals and core difficulties, systematically review the corresponding solutions, and benchmark their performance over multiple widely adopted datasets. In addition, we engage in a discussion of open problems and potential avenues for future research. This paper aims to provide a fresh, comprehensive, and systematic understanding of the challenges and solutions associated with open-environment object detectors, thus catalyzing the development of more solid applications in real-world scenarios.

Citation

If you find our work useful in your research, please consider citing:

@article{liang2024object,
  title={Object Detectors in the Open Environment:Challenges, Solutions, and Outlook}, 
  author={Siyuan Liang and Wei Wang and Ruoyu Chen and Aishan Liu and Boxi Wu and Ee-Chien Chang and Xiaochun Cao and Dacheng Tao},
  eprint={2403.16271},
  year={2024}
}

Menu

• Out of domain Benchmark
  • Data Manipulation
  • Feature Learning
  • Optimization Strategy
• Out of Category Benchmark
  • Discriminant
  • Side Information
  • Arbitrary Information
• Robust Learning Benchmark
  • Adversarial Training Paradigm
  • Model Robust Inference
• Incremental Data Benchmark
  • Replay Based
  • Model Based
  • Regularization Based
  • Optimization Based

Out of Domain Benchmark

Data Manipulation

Paper	Published in	Code/Project
Diversify and Match: A Domain Adaptive Representation Learning Paradigmfor Object Detection	CVPR 2019	Code
Domain randomization for scene-specific car detection and pose estimation	WACV 2019	-
Structured domain randomization: Bridging the reality gap by context-aware synthetic data	ICRA 2019	-
AFAN: augmented feature alignment network for cross-domain object detection	TIP 2021	-
Progressive domain adaptation for object detection	WACV 2020	Code
Harmonizing transferability discriminability for adapting object detectors	CVPR 2020	Code
Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks	ICCV 2017	Code
AsyFOD: An asymmetric adaptation paradigm for few-shot domain adaptive object detection	CVPR 2023	Code
CLIP the gap: A single domain generalization approach for object detection	CVPR 2023	-
Auggan: Cross domain adaptation with gan-based data augmentation	ECCV 2018	-
Domain adaptation for object detection via style consistency	BMVC 2019	-
Target-style-aware unsupervised domain adaptation for object detection	RAL 2021	-
SC-UDA: style content gaps aware unsupervised domain adaptation for object detection	WACV 2022	-

Feature Learning

Paper	Published in	Code/Project
Mega-cda: Memory guided attention for category-aware unsupervised domain adaptive object detection	CVPR 2021	-
Domain adaptive faster R-CNN for object detection in the wild	CVPR 2018	Code
Strong-weak distribution alignment for adaptive object detection	CVPR 2019	Code
Adapting object detectors via selective cross-domain alignment	CVPR 2019	Code
Exploring categorical regularization for domain adaptive object detection	CVPR 2020	Code
Spatial attention pyramid network for unsupervised domain adaptation	ECCV 2020	Code
Multi-adversarial faster-rcnn for unrestricted object detection	ICCV 2019	Code
Domain-adaptive object detection via uncertainty- aware distribution alignment	MM 2020	Code
Deeply aligned adaptation for cross-domain object detection	arXiv 2020	-
Domain adaptive object detection via asymmetric tri-way faster-rcnn	ECCV 2020	-
Adaptive object detection with dual multi-label prediction	ECCV 2020	-
Seeking similarities over differences:Similarity-based domain alignment for adaptive object detection	ICCV 2021	Code
Decoupled adaptation for cross-domain object detection	ICLR 2022	Code
Exploring sequence feature alignment for domain adaptive detection transformers	MM 2021	Code
Informative consistent correspondence mining for cross-domain weakly supervised object detection	CVPR 2021	-
Task-specific inconsistency alignment for domain adaptive object detection	CVPR 2022	Code
Cross-domain detection via graph-induced prototype alignment	CVPR 2020	Code
Cross-domain object detection through coarse-to-fine feature adaptation	CVPR 2020	-
RPN prototype alignment for domain adaptive object detector	CVPR 2021	-
Adapting object detectors with conditional domain normalization	ECCV 2020	Code
Vector-decomposed disentanglement for domain-invariant object detection	ICCV 2021	Code
Instance-invariant domain adaptive object detection via progressive disentanglement	TPAMI 2022	Code
Single-domain generalized object detection in urban scene via cyclic-disentangled self-distillation	CVPR 2022	-

Optimization Strategy

Paper	Published in	Code/Project
Automatic adaptation of object detectors to new domains using self-training	CVPR 2019	-
A robust learning approach to domain adaptive object detection	ICCV 2019	Code
Self-training adversarial background regularization for unsupervised domain adaptive one-stage object detection	ICCV 2019	-
A free lunch for unsupervised domain adaptive object detection without source data	AAAI 2021	-
Category dictionary guided unsupervised domain adaptation for object detection	AAAI 2021	Code
SSAL: synergizing between self-training adversarial learning for domain adaptive object detection	NIPS 2021	-
Multi-view domain adaptive object detection on camera networks	AAAI 2023	-
Exploring object relation in mean teacher for cross domain detection	CVPR 2019	Code
Unbiased teacher for semi-supervised object detection	ICLR 2021	Code
Unbiased mean teacher for cross-domain object detection	CVPR 2021	Code
Cross domain object detection by target-perceived dual branch distillation	CVPR 2022	-
Cross-domain adaptive teacher for object detection	CVPR 2022	Code
Target-relevant knowledge preservation for multi-source domain adaptive object detection	CVPR 2022	-
Multi-source domain adaptation for object detection	ICCV 2021	Code
2pcnet: Two-phase consistency training for day-to-night unsupervised domain adaptive object detection	CVPR 2023	-

Out of Category Benchmark

Discriminant

Paper	Published in	Code/Project
Expanding low-density latent regions for open-set object detection	CVPR 2022	Code
Dropout sampling for robust object detection in open-set conditions	ICRA 2018	-
Evaluating merging strategies for sampling-based uncertainty techniques in object detection	ICRA 2019	-
Vos: Learning what you don’t know by virtual outlier synthesis	ICLR 2022	Code
Towards open world object detection	CVPR 2021	Code
Unknown-aware object detection: Learning what you don’t know from videos in the wild	CVPR 2022	Code
Open-set semi-supervised object detection	ECCV 2022	-
Proposalclip: unsupervised open-category object proposal generation via exploiting clip cues	CVPR 2022	-
Uc-owod: Unknown-classified open world object detection	ECCV 2022	-
Hsic-based moving weight averaging for few-shot open-set object detection	MM 2023	-

Side Information

Paper	Published in	Code/Project
Zero shot detection	TCSVT 2019	Code
Zero-shot object detection with attributes-based category similarity	TCSVT 2020	-
Zero-shot object detection	ECCV 2018	-
Zero-shot object detection with textual descriptions	AAAI 2019	-
Semantics-preserving graph propagation for zero-shot object detection	TIP 2020	-
Polarity loss: Improving visual_semantic alignment for zero-shot detection	TNNLS 2022
From node to graph: Joint reasoning on visual_semantic relational graph for zero-shot detection	WACV 2022	Code
Semantics-guided contrastive network for zero-shot object detection	TPAMI 2022	-
Robust region feature synthesizer for zero-shot object detection	CVPR 2022	-
Generative multi-label zero-shot learning	TPAMI 2023	Code
Zero-shot camouflaged object detection	TIP 2023	-

Arbitrary Information

Paper	Published in	Code/Project
Open-vocabulary object detection using captions	CVPR 2021	-
Open-vocabulary object detection via vision and language knowledge distillation	ICLR 2022	-
Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation	CVPR 2022	-
F-vlm: Open-vocabulary object detection upon frozen vision and language models	ICLR 2023	-
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels	ECCV 2022	Code
PromptDet: Towards Open-vocabulary Detection using Uncurated Images	ECCV 2022	Code
Grounded language-image pre-training	CVPR 2022	Code
Regionclip: Region-based language-image pretraining	CVPR 2022	Code
Detecting everything in the open world: Towards universal object detection	CVPR 2023	Code
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers	CVPR 2023	Code
DetCLIP: Dictionary-enriched visual-concept paralleled pre-training for open-world detection	NIPS 2022	-
DetCLIPv2: Scalable open-vocabulary object detection pre-training via word-region alignment	CVPR 2023	-

Robust Learning Benchmark

Adversarial Training Paradigm

Paper	Published in	Code/Project
Using feature alignment can improve clean average precision and adversarial robustness in object detection	ICIP 2021	-
On the importance of backbone to the adversarial robustness of object detectors	arXiv 2023	-
Adversarial attack and defense of yolo detectors in autonomous driving scenarios	IV 2022	-
Towards adversarially robust object detection	ICCV 2019	-
Robust and accurate object detection via adversarial learning	CVPR 2021	-
Robust and accurate object detection via self knowledge distillation	ICIP 2022	-
Adversarially-aware robust object detector	ECCV 2022	Code
Adversarial intensity awareness for robust object detection	SSRN 2022	-
Towards efficient adversarial training on vision transformers	ECCV 2022	-
Fast is better than free: Revisiting adversarial training	arXiv 2020	-
Class-aware robust adversarial training for object detection	CVPR 2021	-

Model Robust Inference

Paper	Published in	Code/Project
A study of the effect of jpg compression on adversarial images	arXiv 2016	-
Adversarial pixel masking: A defense against physical attacks for pre-trained object detectors	MM 2021	-
Local gradients smoothing: Defense against localized adversarial attacks	WACV 2019	Code
Feature squeezing: Detecting adversarial examples in deep neural networks	arXiv 2017	-
Detection as regression: Certified object detection with median smoothing	NIPS 2020	-
Detectorguard: Provably securing object detectors against localized patch hiding attacks	CCS 2021	Code
Real-time robust video object detection system against physical-world adversarial attacks	TCAD 2023	-
Segment and complete: Defending object detectors against adversarial patch attacks with robust patch detection	CVPR 2022	-
Defending from physically-realizable adversarial attacks through internal over-activation analysis	AAAI 2023	-

Incremental Learning Benchmark

Replay Based

Paper	Published in	Code/Project
Rt-net: replay-and-transfer network for class incremental object detection	Applied Intelligence, 2023	-
Continual detection transformer for incremental object detection	CVPR 2023	Code
Towards open world object detection	CVPR 2021	Code
Continual learning strategy in one-stage object detection framework based on experience replay for autonomous driving vehicle	Sensors 2020	-
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection	ICCV 2023	-
One-shot replay: boosting incremental object detection via retrospecting one object	AAAI 2023	-
Rodeo: Replay for online object detection	arXiv 2020	Code
Utilizing incremental branches on a one-stage object detection framework to avoid catastrophic forgetting	MVA 2022	-
An end-to-end architecture for class-incremental object detection with knowledge distillation	ICME 2019	-
Incremental few-shot instance segmentation	CVPR 2021	Code

Model Based

Paper	Published in	Code/Project
Incremental learning of single-stage detectors with mining memory neurons	ICCC 2018	-
Diode: dilatable incremental object detection	PR 2023	-
Lstd: A low-shot transfer detector for object detection	AAAI 2018	-
Few-shot object detection via feature reweighting	ICCV 2019	Code
Meta-learning-based incremental few-shot object detection	TCSVT 2021	Code
Sylph: A hypernetwork framework for incremental few-shot object detection	CVPR 2022	-
Incremental few-shot object detection	CVPR 2020	-
Few-shot batch incremental road object detection via detector fusion	ICCVW 2021	-
Context-transformer: Tackling object confusion for few-shot detection	AAAI 2020	Code
Incremental learning of object detection with output merging of compact expert detectors	ICoIAS 2021	-
Continual object detection via prototypical task correlation guided gating mechanism	CVPR 2022	-
Dlcft: Deep linear continual fine-tuning for general incremental learning	ECCV 2022	-
Side-tuning: a baseline for network adaptation via additive side networks	ECCV 2020	-
Any-shot object detection	ACCV 2020	-
Incremental-detr: Incremental few-shot object detection via self-supervised learning	AAAI 2023	-
Ow-detr: Open-world detection transformer	CVPR 2022	Code

Regularization Based

Paper	Published in	Code/Project
Exploring object relation in mean teacher for cross-domain detection	CVPR 2019	Code
Unbiased teacher for semi-supervised object detection	ICLR 2021	Code
Unbiased mean teacher for cross-domain object detection	CVPR 2021	Code
Cross domain object detection by target-perceived dual branch distillation	CVPR 2022	-
Cross-domain adaptive teacher for object detection	CVPR 2022	Code
Target-relevant knowledge preservation for multi-source domain adaptive object detection	CVPR 2022	-
Multi-source domain adaptation for object detection	ICCV 2021	Code
2pcnet: Two-phase consistency training for day-to-night unsupervised domain adaptive object detection	CVPR 2023	-
Balanced ranking and sorting for class incremental object detection	ICASSP 2022	-
Incdet: In defense of elastic weight consolidation for incremental object detection	TNNLS 2020	-
ifs-rcnn: An incremental few-shot instance segmenter	CVPR 2022	Code

Optimization Based

Paper	Published in	Code/Project
Towards generalized and incremental few-shot object detection	arXiv 2021	-
Incremental object detection via meta-learning	TPAMI 2021	Code
Fast hierarchical learning for few-shot object detection	IROS 2022	Code