Skip to content
Temperature scaling for neural network calibration
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
include
python
src
utils
CMakeLists.txt
LICENSE
README.md

README.md

libtscaling Temperature scaling for neural network calibration

This library allows calibration of neural networks. See (https://arxiv.org/abs/1706.04599) for detailed problem description and solutions.

Description

Modern deep architectures have good prediction power, but are generally over confident. The idea is to use a validation set after initial training in order to add a distillation phase between training and production (predictions) allowing to calibrate confidence over this unseen set. The output of this library is a distillation temperature to be used for scaling logits before softmax a prediction time (logits should be divided by final temperature before softmax).

Dependencies

Either liblbfgs (liblbfgs-dev on ubuntu) or lbfgspp (http://github.com/yixuan/LBFGSpp) that needs eigen.

Build

Cmake based. If USE_LBFGSPP is turned off, then LBFGS_LIB is mandatory (auto detected in standard pathes); inversely, if LBFGS_LIB is not found, USE_LBFGSPP is forced to on. in details:

  • mkdir build && cd build
  • cmake .. for using system liblbfgs or cmake .. -DLBFGSPP_LIB=path/to/lbfgspp_clone
  • make

gives libtscaling.so

Example usage

API is going to change !!!

important methods:

  • TempScaler::setData(logits, validation_true_labels)

  • TempScaler::calibrate() gives the final temperature

  • CalibrationError(number_of_bins)

  • CalibrationError::setData(confidences, predictions, targets)

  • CalibrationError::ECE() gives expected calibration error

  • CalibrationError::MCE() gives maximum calibration error

  • CalibrationError::percents() gives percent of samples per confidence bin

  • CalibrationError::accuracies() gives accuracies per confidence bin

  • CalibrationError::to_py() output a matploblib'ed python file showing graphs as in (https://arxiv.org/abs/1706.04599)

You can’t perform that action at this time.