OpenOOD v1.5 methods & benchmarks overview

OOD & OSR papers

The methods that are available in OpenOOD v1.5 have the Alias and Group attribute, where Alias is the name of the method, and Group is the category the method falls into (e.g., post-hoc or training method).

Year	Venue	Title	Alias	Group
2016	CVPR	Towards open-set deep networks	OpenMax	post-hoc
2017	ICLR	A simple baseline for detecting out-of-distribution and misclassified examples	MSP	post-hoc
2017	CVPR	Incremental Kernel Null Space Discriminant Analysis for Novelty Detection
2017	ICML	On Calibration of Modern Neural Networks	TempScale	post-hoc
2017	NeurIPS	Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
2018	ICLR	Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
2018	ICLR	Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks	ODIN	post-hoc
2018	CVPR	Hierarchical Novelty Detection for Visual Object Recognition
2018	ICML	Open Category Detection with PAC Guarantees
2018	ECCV	Open Set Learning with Counterfactual Images
2018	ECCV	Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers
2018	NeurIPS	Out-of-Distribution Detection using Multiple Semantic Label Representations
2018	NeurIPS	Generative Probabilistic Novelty Detection with Adversarial Autoencoders
2018	NeurIPS	Deep Anomaly Detection Using Geometric Transformations
2018	NeurIPS	A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks	MDS / MDSEns	post-hoc
2018	arXiv	Learning confidence for out-of-distribution detection in neural networks	ConfBranch	training
2019	ICLR	Deep Anomaly Detection with Outlier Exposure	OE	training (w/ outliers)
2019	ICLR	Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness	StyleAugment	data augmentation
2019	CVPR	C2AE: Class Conditioned Auto-Encoder for Open-Set Recognition
2019	CVPR	Large-Scale Long-Tailed Recognition in an Open World
2019	CVPR	Classification-Reconstruction Learning for Open-Set Recognition
2019	CVPR	Deep Transfer Learning for Multiple Class Novelty Detection
2019	ICML	Using Pre-Training Can Improve Model Robustness and Uncertainty
2019	ICCV	Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy	MCD	training (w/ outliers)
2019	NeurIPS	Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty	RotPred	training
2019	NeurIPS	Likelihood Ratios for Out-of-Distribution Detection
2019	NeurIPS	A Simple Baseline for Bayesian Uncertainty in Deep Learning
2019	NeurIPS	On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks
2020	ICLR	Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models
2020	ICLR	RaPP: Novelty Detection with Reconstruction along Projection Pathway
2020	ICLR	Robust anomaly detection and backdoor attack detection via differential privacy
2020	ICLR	Novelty Detection Via Blurring
2020	ICLR	AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty	AugMix	data augmentation
2020	CVPR	Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data	G-ODIN	training
2020	CVPR	Conditional Gaussian Distribution Learning for Open Set Recognition
2020	CVPR	Generative-Discriminative Feature Representations for Open-Set Recognition
2020	CVPR	Deep Residual Flow for Out of Distribution Detection
2020	ICML	Detecting Out-of-Distribution Examples with Gram Matrices	Gram	post-hoc
2020	ECCV	Adversarial Robustness on In- and Out-Distribution Improves Explainability
2020	ECCV	Utilizing Patch-level Category Activation Patterns for Multiple Class Novelty Detection
2020	ECCV	Hybrid Models for Open Set Recognition
2020	NeurIPS	Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples
2020	NeurIPS	Why Normalizing Flows Fail to Detect Out-of-Distribution Data
2020	NeurIPS	Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
2020	NeurIPS	Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features
2020	NeurIPS	Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
2020	NeurIPS	Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
2020	NeurIPS	Energy-based Out-of-distribution Detection	EBO	post-hoc
2020	NeurIPS	CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances	CSI	training
2020	NeurIPS	RandAugment: Practical Automated Data Augmentation with a Reduced Search Space	RandAugment	data augmentation
2020	AAAI	Self-Supervised Learning for Generalizable Out-of-Distribution Detection
2020	AAAI	Uncertainty-Aware Deep Classifiers using Generative Models
2021	arXiv	A simple fix to mahalanobis distance for improving near-ood detection	RMDS	post-hoc
2021	ICLR	Multiscale Score Matching for Out-of-Distribution Detection
2021	ICLR	SSD: A Unified Framework for Self-Supervised Outlier Detection
2021	CVPR	MOOD: Multi-Level Out-of-Distribution Detection
2021	CVPR	MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space	MOS	training
2021	CVPR	Natural Adversarial Examples
2021	CVPR	Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces
2021	CVPR	Learning Deep Classifiers Consistent With Fine-Grained Novelty Detection
2021	CVPR	Counterfactual Zero-Shot and Open-Set Visual Recognition
2021	CVPR	Learning Placeholders for Open-Set Recognition
2021	ICML	Learning Bounds for Open-Set Learning
2021	ICCV	CODEs: Chamfer Out-of-Distribution Examples Against Overconfidence Issue
2021	ICCV	Conditional Variational Capsule Network for Open Set Recognition
2021	ICCV	Energy-Based Open-World Uncertainty Modeling for Confidence Calibration
2021	ICCV	Semantically Coherent Out-of-Distribution Detection	UDG	training (w/ outliers)
2021	ICCV	OpenGAN: Open-Set Recognition via Open Data Generation	OpenGAN	post-hoc
2021	ICCV	The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization	DeepAugment	data augmentation
2021	NeurIPS	ReAct: Out-of-distribution Detection With Rectified Activations	ReAct	post-hoc
2021	NeurIPS	On the Importance of Gradients for Detecting Distributional Shifts in the Wild	GradNorm	post-hoc
2021	NeurIPS	Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
2021	NeurIPS	Exploring the Limits of Out-of-Distribution Detection
2021	NeurIPS	Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models
2021	AAAI	Open-Set Recognition with Gaussian Mixture Variational Autoencoders
2021	TPAMI	Adversarial reciprocal points learning for open set recognition	ARPL	training
2022	ICLR	Open-Set Recognition: A Good Closed-Set Classifier is All You Need
2022	ICLR	Igeood: An Information Geometry Approach to Out-of-Distribution Detection
2022	ICLR	Revisiting flow generative models for Out-of-distribution detection
2022	ICLR	A Statistical Framework for Efficient Out of Distribution Detection in Deep Neural Networks
2022	ICLR	VOS: Learning What You Don't Know by Virtual Outlier Synthesis	VOS	training
2022	CVPR	ViM: Out-of-Distribution With Virtual-Logit Matching	VIM	post-hoc
2022	CVPR	PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures	PixMix	data augmentation
2022	CVPR	Deep Hybrid Models for Out-of-Distribution Detection
2022	ICML	POEM: Out-of-Distribution Detection with Posterior Sampling
2022	ICML	Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities
2022	ICML	Scaling out-of-distribution detection for real-world settings	MLS, KLM	post-hoc
2022	ICML	Out-of-Distribution Detection with Deep Nearest Neighbors	KNN	post-hoc
2022	ICML	Mitigating Neural Network Overconfidence with Logit Normalization	LogitNorm	training
2022	ECCV	DICE: Leveraging Sparsification for Out-of-Distribution Detection	DICE	post-hoc
2022	ECCV	Out-of-Distribution Detection with Boundary Aware Learning
2022	ECCV	Out-of-Distribution Detection with Semantic Mismatch under Masking
2022	ECCV	Data Invariants to Understand Unsupervised Out-of-Distribution Detection
2022	ECCV	Towards Accurate Open-Set Recognition via Background-Class Regularization
2022	NeurIPS	Out-of-Distribution Detection via Conditional Kernel Independence Model
2022	NeurIPS	Your Out-of-Distribution Detection Method is Not Robust!
2022	NeurIPS	Boosting Out-of-distribution Detection with Typical Features
2022	NeurIPS	RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection	RankFeat	post-hoc
2022	NeurIPS	Using Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness	RegMixup	data augmentation
2022	NeurIPS	Watermarking for Out-of-distribution Detection
2022	NeurIPS	Density-driven Regularization for Out-of-distribution Detection
2022	NeurIPS	Delving into Out-of-Distribution Detection with Vision-Language Representations
2022	NeurIPS	Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE
2023	WACV	Mixture Outlier Exposure: Towards Out-of-Distribution Detection in Fine-Grained Environments	MixOE	training (w/ outliers)
2023	ICLR	Harnessing Out-Of-Distribution Examples via Augmenting Content and Style
2023	ICLR	The Tilted Variational Autoencoder: Improving Out-of-Distribution Detection
2023	ICLR	Out-of-distribution Detection with Implicit Outlier Transformation
2023	ICLR	Extremely Simple Activation Shaping for Out-of-Distribution Detection	ASH	post-hoc
2023	ICLR	Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy	SHE	post-hoc
2023	ICLR	How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?	CIDER	training
2023	ICLR	Non-parametric Outlier Synthesis	NPOS	training
2023	CVPR	GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection	GEN	post-hoc

OOD benchmarks

Overview

ID	near-OOD	far-OOD	Covariate-shifted ID	Outliers for training
CIFAR-10	CIFAR-100, Tiny ImageNet-200	MNIST, SVHN, Textures, Places365	N/A	Tiny ImageNet-597
CIFAR-100	CIFAR-10, Tiny ImageNet-200	MNIST, SVHN, Textures, Places365	N/A	Tiny ImageNet-597
ImageNet-200	SSB-hard, NINCO	iNaturalist, Textures, OpenImage-O	ImageNet-200-C, ImageNet-200-R, ImageNet-200-V2	ImageNet-800
ImageNet-1K	SSB-hard, NINCO	iNaturalist, Textures, OpenImage-O	ImageNet-C, ImageNet-R, ImageNet-V2	N/A

Breakdown

CIFAR-10, CIFAR-100, Tiny ImageNet-200, MNIST, SVHN, Places365, and iNaturalist are commonly-used datasets. Here we describe the other less common ones.

• Tiny ImageNet-597: This is the dataset we use as outliers for CIFAR experiments. Basically we filter out many categories from ImageNet-1K to avoid overlap with test OOD data, resulting in 597 categories left. Then we apply the same processing as ImageNet -> Tiny ImageNet to create this Tiny ImageNet-597. Please see our paper or this post for details.
• Textures: Also known as Describable Textures Dataset (DTD).
• ImageNet-C: A variant of ImageNet with with common corruptions applied (e.g., motion blurring).
• ImageNet-R: A variant of ImageNet that has styles different than natural images (e.g., art paintings). It includes 200 categories of ImageNet.
• ImageNet-V2: A variant of ImageNet that has resampling shift.
• ImageNet-200: The 200-class subset of ImageNet-1K. The categories are the same as those in ImageNet-R.
• ImageNet-200-C/R/V2: The corresponding 200-class version of ImageNet-C/R/V2.
• ImageNet-800: The 800-class subset of ImageNet-1K that is disjoint with ImageNet-200. We use it as the outlier dataset for ImageNet-200 experiments.
• SSB-hard: The hard split from SSB, which is curated by selecting "hard" categories from ImageNet-21K based on WordNet hierarchy.
• NINCO: A recent OOD dataset that is manually inspected to be strictly OOD w.r.t. ImageNet-1K.