Implementations of Uncertainty Quantification Methods for Neural Networks

This repository collects available implementations for uncertainty quantification and calibration methods in neural networks. The list mainly links to resources which are provided by the authors of the corresponding approaches or to other collections of implementations.

Method Type	Approch	Paper(s)	Authors official Repository	Framework	Link to Repository
Calibration	Temperature Scaling	C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, "On calibration of modern neural networks", International Conference on Machine Learning (2017). [paper]	yes	PyTorch	https://github.com/gpleiss/temperature_scaling
Single Deterministic Neural Network	Misclassification and Out-of-Distribution detection based on the softmax output.	D. Hendrycks and K. Gimpel. "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136 (2016). [paper]	yes	TensorFlow	https://github.com/hendrycks/error-detection
Single Deterministic Neural Network & Calibration	Implementation of Odin and Temperature Scalling for out-of-distribution detection.	S. Liang, Y. Li and R. Srikant, "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks", International Conference on Learning Representations (2018). [paper]	yes	PyTorch	https://github.com/facebookresearch/odin
Single Deterministic Neural Network	Outlier exposure - Learn heuristic for out-of-distribution detection from out-of-distribution examples.	Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. "Deep anomaly detection with outlier exposure." arXiv preprint arXiv:1812.04606 (2018). [paper]	yes	PyTorch	https://github.com/hendrycks/outlier-exposure
Single Deterministic Neural Network	Deep Mahalanobis Detector for Out-of-Dsitribution detection	K. Lee, K. Lee, H. Lee, J. Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”, NeurIPS 2018. [paper]	yes	PyTorch	https://github.com/pokaxpoka/deep_Mahalanobis_detector
Single Deterministic Neural Network	Evidential Neural Networks for Classification tasks	M. Sensoy, L. Kaplan, and M. Kandemir, “Evidential deep learning to quantify classification uncertainty” , NeurIPS 2018. [paper]	partially	Tensorflow & PyTorch	https://github.com/dougbrion/pytorch-classification-uncertainty https://muratsensoy.github.io/uncertainty.html
Single Deterministic Neural Network	Dirichlet Prior Networks	A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks” NeurIPS 2018. [paper] A. Malinin and M. Gales, “Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness” </ i>, NeurIPS 2019. [paper]	yes	PyTorch	https://github.com/KaosEngineer/PriorNetworks
Bayesian Neural Network	Bayes By Backprop (BBB)	C. Blundell, J. Cornebise, K. Kavukcuoglu & D. Wierstra "Weight Uncertainty in Neural Network", ICML 2015. [paper]	no	PyTorch	https://github.com/nitarshan/bayes-by-backprop https://github.com/ThirstyScholar/bayes-by-backprop
Bayesian Neural Network	Example implementation of Multiplicative Normalizing Flows	C. Louizos and M. Welling, "Multiplicative Normalizing Flows for Variational Bayesian Neural Networks", ICML 2017. [paper]	no	TensorFlow & PyTorch	https://github.com/AMLab-Amsterdam/MNF_VBNN https://github.com/janosh/torch-mnf
Bayesian Neural Network	Monte Carlo Dropout	Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." ICML 2016. [paper]	yes	TensorFlow	https://github.com/yaringal/DropoutUncertaintyExps
Bayesian Neural Network	VOGN Optimizer and presentation of Bayesian principles in neural networks.	M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal and A. Y. Srivastava, "Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam", ICML 2018. [paper]	yes	PyTorch	https://github.com/team-approx-bayes/dl-with-bayes
Bayesian Neural Network	Provides facilities to easily train your PyTorch neural network models using variational inference.	-	no	PyTorch	https://github.com/ctallec/pyvarinf
Bayesian Neural Network	Stochastic Weight Averaging-Gaussian (SWAG)	W. J. Maddox, et al. "A simple baseline for bayesian uncertainty in deep learning." NeurIPS 2019: 13153-13164. [paper]	yes	PyTorch	https://github.com/wjmaddox/swa_gaussian
Bayesian Neural Network	Deterministic Variational Inference	A. Wu, S. Nowozin, E. Meeds, R. E. Turner, J. M. Hernandez-Lobato, A. L. Gaunt, Deterministic Variational Inference for Robust Bayesian Neural Networks, ICLR 2019. [paper]	partially	TensorFlow & PyTorch	https://github.com/Microsoft/deterministic-variational-inference https://github.com/markovalexander/DVI
Bayesian Neural Network	Laplace Approximation with four different approximations of the curvature	Estimating Model Uncertainty of Neural Networks in Sparse Information Form J. Lee, M. Humt, J. Feng, R. Triebel, ICLR 2020. [paper] Bayesian Optimization Meets Laplace Approximation for Robotic Introspection M. Humt, J. Lee, R. Triebel, IROS 2020 Workshop. [paper] "Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry", K. Shinde, J. Lee, M. Humt, A. Sezgin, R. Triebel, ICLR 2020 Workshop. [paper]	yes	PyTorch	https://github.com/DLR-RM/curvature
Ensembles	Deep Ensembles	B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles” NeurIPS 2017. [paper]	no	TensorFlow	https://github.com/vvanirudh/deep-ensembles-uncertainty https://github.com/axelbrando/Mixture-Density-Networks-for-distribution-and-uncertainty-estimation https://github.com/Kyushik/Predictive-Uncertainty-Estimation-using-Deep-Ensemble
Collection	Collection containing Deep Ensembles Stochastic Weight-Averaging Gaussian Kronecker-Factored Laplace Approximation	A. Ashukha, A. Lyzhov, D. Molchanov and D. Vetrov, "Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning", ICLR 2020	yes	PyTorch	https://github.com/bayesgroup/pytorch-ensembles
Collection	Collection containing Deep Ensembles Bayes by Backprop Monte Carlo Dropout Temperature scaling	-	no	PyTorch	https://github.com/cpark321/uncertainty-deep-learning
Collection	Collection containing Bayes by Backprop Monte Carlo Dropout Stochastic gradient Langevin dynamics Kronecker-Factored Laplace Approximation Stochastic Gradient Hamiltonian Monte Carlo	-	no	PyTorch	https://github.com/JavierAntoran/Bayesian-Neural-Networks