Uncertainty Toolbox: a Python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

A Library for Uncertainty Quantification.

A collection of research and application papers of (uncertainty) calibration techniques.

A toolkit for visualizations in materials informatics.

Calibration library and code for the paper: Verified Uncertainty Calibration. Ananya Kumar, Percy Liang, Tengyu Ma. NeurIPS 2019 (Spotlight).

Code to accompany the paper 'Improving model calibration with accuracy versus uncertainty optimization'.

(ECCV 2022) BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks

Code for evaluating uncertainty estimation methods for Transformer-based architectures in natural language understanding tasks.

A project to train your model from scratch or fine-tune a pretrained model using the losses provided in this library to improve out-of-distribution detection and uncertainty estimation performances. Calibrate your model to produce enhanced uncertainty estimations. Detect out-of-distribution data using the defined score type and threshold.

Scoring rules like the Brier Score (Mean Squared Error, Quadratic Score) and Log Loss (Cross-Entropy, Negative Log-Likelihood, Logarithmic Score) can favor incorrect predictions. To address this limitation, the Probabilistic Brier Score (PBS) and Probabilistic Logarithmic Loss (PLL) have been introduced for probabilistic classifiers.

Source code for our paper: "LoGU: Long-form Generation with Uncertainty Expressions".

Service to examine data processing pipelines (e.g., machine learning or deep learning pipelines) for uncertainty consistency (calibration), fairness, and other safety-relevant aspects.

Calibration of Few-Shot Classification Tasks: Mitigating Misconfidence from Distribution Mismatch, IEEE Access vol.10

Truth Discovery Promotes Uncertainty Calibration of DNNs (UAI 2021)