Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brier Score decomposition: REL, RES, UNC #232

Closed
dplarson opened this issue Oct 24, 2019 · 3 comments · Fixed by #202
Closed

Brier Score decomposition: REL, RES, UNC #232

dplarson opened this issue Oct 24, 2019 · 3 comments · Fixed by #202
Labels
metrics Issue pertains to metrics calculation
Milestone

Comments

@dplarson
Copy link
Contributor

Opening this issue here (rather than the website repo) since this is more a question of code implementation that metric definition:

The Brier Score (BS) can be decomposed into three components: reliability (REL), resolution (RES) and uncertainty (UNC), where BS = REL - RES + UNC. To compute the REL and RES components, you find the K unique forecasts in the N forecasts provided (K <= N). Since the probabilistic forecasts can be provided to SolarArbiter as floating point numbers of arbitrary precision, this brings up the question: how do you determine a (finite) set of unique forecasts?

One approach is converting all forecasts to a pre-defined precision (e.g. one or two decimal places). This is seemingly the approach taken in the original paper [1], which states that the forecasts "can assume only a finite set of S distinct values". Also, in [1], the examples provided use 1-decimal precision forecasts (0.0, 0.1, 0.2, ..., 1.0), with some (apparent) rounding of intermediate calculations.

[1] Murphy (1973) "A New Vector Partition of the Probability Score", doi: https://doi.org/10.1175/1520-0450(1973)012%3C0595:ANVPOT%3E2.0.CO;2

Question

  • How should SolarArbiter determine the (finite) sets of unique forecasts for the REL and RES metrics?
@dplarson
Copy link
Contributor Author

dplarson commented Oct 24, 2019

My suggestion is that we should approximate the probabilities [-] to two decimal precision (e.g. 0.2354 => 0.24) for the REL and RES calculations only, but I'm not sure yet on the best approach to approximating the probabilities (since we still want the probabilities to sum to 1.00 and it's possible that simple rounding will result in the sum coming out as, e.g., 0.99 or 1.01).

@wholmgren
Copy link
Member

wholmgren commented Oct 24, 2019

Do we need to worry about the number of bins as vs. the number of forecasts/observations? For example, if N < 1000, bin by tenths, otherwise bin by hundredths?

In any case, we could follow the formulation in Stephenson et. al. "Two Extra Components in the Brier Score Decomposition".

@wholmgren wholmgren added the metrics Issue pertains to metrics calculation label Oct 24, 2019
@wholmgren wholmgren added this to the 1.0 beta 2 milestone Oct 24, 2019
@dplarson
Copy link
Contributor Author

That's a good point and I agree we'd probably want to explicitly set some sort of convention on number of bins versus number of forecasts/evaluation. (At the very least, define what to do if there are very few forecasts/observations, e.g., << 100.)

Also, thanks for sharing that paper (I'm reading through it now).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
metrics Issue pertains to metrics calculation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants