-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
core(scoring): loosen metric thresholds #5092
Conversation
ced4151
to
e98532f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
does this affect the decisions that went into #5083?
for better or for worse, the score thresholds/curves of the metrics themselves were not considered at all as part of weighting committee rulings, so this doesn't affect it :) |
@patrickhulce what does PODR mean? |
@kaycebasques it stands for Point Of Diminishing Returns It's roughly a score of ~97 and is the point on the log normal probability curve where the derivative reaches its peak. Basically, going farther past that point will yield minimal increases to the score. |
FYI for future code archaeologists: PODR is not necessarily ~97, it varies based on what the median is set as. Everything else @patrickhulce said is true though. |
Yes, future code archaeologists please do not interpret "roughly a score of ~97" to mean that it in any way is trying to target a score of 97. It just so happened that for many of our curves the PODR is 99-95, so roughly ~97 felt appropriate to say. Everything else @connorjclark said is true 😉 |
As we just discussed, the new scoring thresholds based on 50/75 were making the requirements for 99/100 too unachievable. The root issue was that our PODR was not considered with our policy. The new policy instead...
Median === 75th percentile
PODR = 95th percentile
This makes a 100 roughly equal to the 98th percentile and keeps the green/orange boundary of 75 to the ~87th percentile which is pretty close to our old policy. Checkout the new scoring calculator sheet https://docs.google.com/spreadsheets/d/1Cxzhy5ecqJCucdf1M0iOzM8mIxNc7mmx107o5nj38Eo/edit#gid=0 to play around with the values.