Add Expected Calibration Error

[edwardclem]

🚀 Feature

A new metric computing the Expected Calibration Error (ECE) metric from Naeini et al 2015. Useful for determining if a classifier's softmax probability scores are well-calibrated and represent reasonable probabilities.

Motivation

I ran into this metric after seeing Guo et al 2017, which discussed how very large and deep networks are vulnerable to calibration issues (i.e. are systemically over- or under- confident), and suggest temperature scaling as a method for producing reasonably calibrated softmax outputs using a validation dataset. I've implemented a simple version of this metric using the old PyTorch Lightning Metric API, mostly based off of @gpleiss's PyTorch nn.Module implementation of the metric here.

Additional context

There was an interesting preprint discussing alternatives to ECE that also might be worth integrating - the main criticism being that ECE can be reductive in multi-class settings by not considering all probabilities produced by the model and only looking at the probability of the predicted class (i.e. the highest predicted probability). I'd have to think more about this, but it could also be worth adding.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

@edwardclem sounds like a great addition! want to send a PR?

[edwardclem]

Sure! Let me write up some test cases first. Is there any documentation I should look at about this library's test practices or should I just look at some of the other classification metric tests?

[SkafteNicki]

@edwardclem how is it going here?

[edwardclem]

I have the calibration error metric working and tested with the L1, L2, and max norms matching the behavior of the sklearn PR here. There appears to be an implementation difference in the bias term correction from this paper that causes those tests to fail, and I'm checking with the sklearn devs to see which one is correct. I've confirmed that if I change my torch code to exactly match the numpy code the behavior is the same, but I think there might be a small mistake in the sklearn implementation. Either way, I'm happy to open a PR so the review can start. If we want to move ahead with this feature, I can just remove the bias correction term and commit the rest. What do you think?

[SkafteNicki]

Hi @edwardclem, sounds like a great plan to me (sorry for not getting back to you sooner).
Please feel free to open a PR :]