You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Opening this issue here (rather than the website repo) since this is more a question of code implementation that metric definition:
The Brier Score (BS) can be decomposed into three components: reliability (REL), resolution (RES) and uncertainty (UNC), where BS = REL - RES + UNC. To compute the REL and RES components, you find the K unique forecasts in the N forecasts provided (K <= N). Since the probabilistic forecasts can be provided to SolarArbiter as floating point numbers of arbitrary precision, this brings up the question: how do you determine a (finite) set of unique forecasts?
One approach is converting all forecasts to a pre-defined precision (e.g. one or two decimal places). This is seemingly the approach taken in the original paper [1], which states that the forecasts "can assume only a finite set of S distinct values". Also, in [1], the examples provided use 1-decimal precision forecasts (0.0, 0.1, 0.2, ..., 1.0), with some (apparent) rounding of intermediate calculations.
My suggestion is that we should approximate the probabilities [-] to two decimal precision (e.g. 0.2354 => 0.24) for the REL and RES calculations only, but I'm not sure yet on the best approach to approximating the probabilities (since we still want the probabilities to sum to 1.00 and it's possible that simple rounding will result in the sum coming out as, e.g., 0.99 or 1.01).
Do we need to worry about the number of bins as vs. the number of forecasts/observations? For example, if N < 1000, bin by tenths, otherwise bin by hundredths?
In any case, we could follow the formulation in Stephenson et. al. "Two Extra Components in the Brier Score Decomposition".
That's a good point and I agree we'd probably want to explicitly set some sort of convention on number of bins versus number of forecasts/evaluation. (At the very least, define what to do if there are very few forecasts/observations, e.g., << 100.)
Also, thanks for sharing that paper (I'm reading through it now).
Opening this issue here (rather than the website repo) since this is more a question of code implementation that metric definition:
The Brier Score (BS) can be decomposed into three components: reliability (REL), resolution (RES) and uncertainty (UNC), where BS = REL - RES + UNC. To compute the REL and RES components, you find the
K
unique forecasts in theN
forecasts provided (K <= N
). Since the probabilistic forecasts can be provided to SolarArbiter as floating point numbers of arbitrary precision, this brings up the question: how do you determine a (finite) set of unique forecasts?One approach is converting all forecasts to a pre-defined precision (e.g. one or two decimal places). This is seemingly the approach taken in the original paper [1], which states that the forecasts "can assume only a finite set of S distinct values". Also, in [1], the examples provided use 1-decimal precision forecasts (0.0, 0.1, 0.2, ..., 1.0), with some (apparent) rounding of intermediate calculations.
[1] Murphy (1973) "A New Vector Partition of the Probability Score", doi: https://doi.org/10.1175/1520-0450(1973)012%3C0595:ANVPOT%3E2.0.CO;2
Question
The text was updated successfully, but these errors were encountered: