Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
src
 
 
 
 
 
 
 
 

libtscaling Temperature scaling for neural network calibration

This library allows calibration of neural networks. See (https://arxiv.org/abs/1706.04599) for detailed problem description and solutions.

Description

Modern deep architectures have good prediction power, but are generally over confident. The idea is to use a validation set after initial training in order to add a distillation phase between training and production (predictions) allowing to calibrate confidence over this unseen set. The output of this library is a distillation temperature to be used for scaling logits before softmax a prediction time (logits should be divided by final temperature before softmax).

Dependencies

Either liblbfgs (liblbfgs-dev on ubuntu) or lbfgspp (http://github.com/yixuan/LBFGSpp) that needs eigen.

Build

Cmake based. If USE_LBFGSPP is turned off, then LBFGS_LIB is mandatory (auto detected in standard pathes); inversely, if LBFGS_LIB is not found, USE_LBFGSPP is forced to on. in details:

  • mkdir build && cd build
  • cmake .. for using system liblbfgs or cmake .. -DLBFGSPP_LIB=path/to/lbfgspp_clone
  • make

gives libtscaling.so

Example usage

API is going to change !!!

important methods:

  • TempScaler::setData(logits, validation_true_labels)

  • TempScaler::calibrate() gives the final temperature

  • CalibrationError(number_of_bins)

  • CalibrationError::setData(confidences, predictions, targets)

  • CalibrationError::ECE() gives expected calibration error

  • CalibrationError::MCE() gives maximum calibration error

  • CalibrationError::percents() gives percent of samples per confidence bin

  • CalibrationError::accuracies() gives accuracies per confidence bin

  • CalibrationError::to_py() output a matploblib'ed python file showing graphs as in (https://arxiv.org/abs/1706.04599)

About

Temperature scaling for neural network calibration

Topics

Resources

License

Releases

No releases published

Packages

No packages published