Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about Line 63 #18

Open
ltbyron opened this issue Jan 25, 2020 · 1 comment
Open

Question about Line 63 #18

ltbyron opened this issue Jan 25, 2020 · 1 comment

Comments

@ltbyron
Copy link

ltbyron commented Jan 25, 2020

The current code optimize temperature parameter with respect to nll_criterion, is it more reasonable to optimize with respect to ece_criterion in order to obtain a well-calibrated model?

Thanks!

@mikeFore4
Copy link

NLL and brier scores are both "proper scoring rules" for calibration while ECE is not. This means that these losses (NLL & brier) are minimized if and only if the model recovers the true ground truth probability distribution (this is mentioned in the temperature scaling paper and discussed more in depth in "Elements of Statistical Learning" by Friedman).

ECE is not a proper scoring rule because, as explained in Can You Trust Your Model's Uncertainty? "there exist trivial solutions which yield optimal scores; for example, returning the marginal probability p(y) for every instance will yield perfectly calibrated but uninformative predictions."

Bottomline: ECE is useful because it is easily interpreted, but often shouldn't be used to train you network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants