Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brier scores have stayed the same despite many forecasts' resolving #5

Open
patbl opened this issue Aug 27, 2023 · 3 comments
Open

Comments

@patbl
Copy link

patbl commented Aug 27, 2023

I created an account and imported my Prediction Book history. I think that since then, the Brier scores displayed by this UI element have stayed the same:

image

My memory is that 0.36 was displayed for "Last three months" even when I initially hadn't made any predictions yet (and all my Prediction Book predictions had resolved more than three months ago).

@adam-binks
Copy link
Contributor

Hi @patbl - I think this was caused by the "last 3 months" looking at when PredictionBook questions were imported, not when they were actually resolved on PredictionBook. I've pushed a fix for this now in this commit.

I think this would cause your "Last 3 months" Brier to stay static if you have lots of "All time" resolved predictions, because a few new predictions won't change it very much. Could you see if it's now fixed?

@patbl
Copy link
Author

patbl commented Aug 29, 2023

Would you expect your change to affect predictions that had previously been imported from Prediction Book, or just new imports?

The past-three-months and all-time scores now differ a little bit, but that could be because I've been making and resolving a lot of predictions recently:

image

@patbl
Copy link
Author

patbl commented Dec 17, 2023

My date chart still goes back only to August—is this expected to be the case after your fix?

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants