Match Result Predictions #1478
8 participants
|
These predictions are probably going to be very incorrect for Playoffs. Maybe those should be disabled until we have a better scoring model? |
|
If I'm correct, this new xOPR model relies on OPRs from previous events? I'd suggest a different way of approaching this issue. If an OPR from a previous event is used as an input into the function, it may cause predictions to be largely different to what one would expect. For example, a Team may make a major modification to their robot, which would cause any previous OPRs to become irrelevant when calculating their predicted score, whether it be for better or for worse. I would recommend giving different events a different 'weighting' depending on how long ago they were played, as driver experience and confidence can play a huge part in the effectiveness of a robot's performance or contribution to the alliance. I am aware there is currently a weighting on amount of matches played, but perhaps 'Week X' events could have a higher weighting than 'Week X-1' events. Just my $0.02 |
|
@JacisNonsense The contribution of previous event OPRs decays over the course of the event. I agree this isn't necessarily ideal, but it's used to keep the OPR stable when there are few matches played. I'm open to better ideas for keeping OPR stable. Edit: xOPR only uses the OPR from a team's last event, not any events before that. |
|
Okay, the current calculation of xOPR doesn't do what I want it to. Stay tuned. |
|
I played around with a few new OPR algorithms and came up with two: xOPR (eXtended OPR) uses a team's OPR from their last event (or any other source for OPR initialization). Instead of ignoring unplayed matches when computing OPR, xOPR predicts the score of the unplayed matches (a sum of an alliance's OPRs) using the initialization source. The effect is a more constrained solve for M*x=s. However, this leads to a team's past OPR significantly affecting their xOPR calculation until their last match is played. The solution to this is ixOPR. ixOPR (Iterative xOPR) does the same computation as xOPR, but then iterates using the resulting xOPR values as the initialization source (instead of past events). The number of iterations was experimentally chosen to be 2; iterating until convergence was shown to be a bad idea and can take hundreds of iterations, and 2 seems to work well enough. Here are some graphs showing the results. Notice that the initial ixOPR estimate has a lower error than the OPR estimate due to using past event data. The one exception is 2016scmb, which was the first competition of the year, so no team had an initial OPR. In all cases, xOPR and ixOPR is much more stable than OPR. However, ixOPR converges much more quickly to the final OPR than xOPR.
|
|
Here are graphs showing the number of correct predictions compared to the total number of matches within a win margin bracket. The x-axis labels are a bit funky (multiply them by 10 to get the actual score margin)
|
|
I'm concerned that predicted losses may be demotivating to teams and their friends. It's also clutter for most people. How about hiding it except when turned on in a user setting? |
|
+1 to @1fish2's suggestion about making it a user setting. Also, I would suggest graying the text out so that it's more clear that the score is not final. (Maybe ditch the parentheses and italics but keep the asterisk too? Just aesthetic preferences on my part.) |
|
I agree it should be a user setting which is off by default. I also would like to see some kind of clear demarcation of the last match which posted actual results. It will be harder to tell where in the schedule real matches are with all boxes filled by default. I believe the default for a tied match score is to not bold either score, so there needs to be a clear difference between final and predicted scores, even adjacent to ties. |
|
I wholly agree with @1fish2 on demotivation, but many people I have talked to think that the information is nice to have for those who want it. There is discussion in the Facebook group about how to make it more hidden. https://www.facebook.com/groups/moardata/1100696009950693/ |
|
Maybe in our disclaimer, we need to add Gamesense's line: don't like our predictions? Go out there and prove us wrong! |
|
For stats lovers, wouldn't you rather see the predictions in a separate column alongside the actual scores? |
It's computationally expensive to show predictions of matches that have been played using only matches up to the one before a particular match without saving predictions in the database. I want to keep things simple and not touch the database right now, but it's something we can add in the future if we decide to keep predictions around. |
|
Interesting graphs from MSC. ixOPR is never off by the final OPR by more than ~6. I think because all the teams are playing at such a high level, they are super consistent and their last event's OPRs are quite accurate at predicting their current event OPR.
|
|
How's this? @1fish2 @bvisness @Nuttyman54 The '?' will link to something like the current OPR page
|
|
Can we put the toggle on the line with "Qualification Results" or otherwise inside of the "Results" tab? It's only about the results tab, so I feel like that's where we should put it. It feels weird how it's pushing the page down right now. |
|
Yeah I was having trouble deciding where to put it. I put it above the tabs because if we support ranking predictions in the future, it may make more sense to have one global control instead of one on each tab. Also, when there are playoff matches, having the button in line with "Qualification Results" looks awkward. Any other suggestions? |
|
I think we should take a stab at redesigning event, team, and match pages On Sat, Apr 16, 2016 at 12:10 PM Eugene Fang notifications@github.com
|
|
|
Who's the target audience for predictions? What will they do with them? For stats-lovers, won't they need to see all the predictions alongside actual scores (as they arrive) to get feedback on accuracy? If that's expensive to compute, that's a separate issue which might be addressed by computing the predictions in the client, or computing predictions only for users who turned them on, or something. For most spectators, are they interested in OPRs and predictions? Will they understand them? I've heard people in the stands talk as if a higher OPR was why one team picked another. Hopefully teams use multiple criteria to form an alliance with complementary skills rather than using any one score. For others? |
|
@1fish2 I sense a lot of hesitancy about adding predictions, and I think your concerns are justified. I don't know what the exact target audience is, but people have requested it "since Spyder does it" there seems to be general interest around it (gauging from the Facebook page/group and people I've talked to), but then again the "hard core" FIRSTers are probably I'm hearing, not the general audience. Personally, I think it provides an opportunity for spectators to look for close high scoring matches that may be interesting, or for a team to be more informed about their expected final seed as well as the absolute highest/lowest they can seed (once we implement predictive rankings) so they can more appropriately plan for alliance selections. |
|
This conversation reminds me a lot of the initial hesitance we had with adding OPRs to the site at all. It's important that The Blue Alliance remain a tool that serves as broad of an audience as possible. We should make decisions that draw as many people in to FIRST as possible, and keep the site easy and clear for FIRSTers to share with a not-yet-FIRST audience. I also agree there is value in digging into some of these stats, and thinking about how we can build powerful scouting tools at competitions. We've seen the pages for Events, Teams and Matches get much more complicated over time as we've added more (useful) information to them, but information that isn't going to be needed by every visitor to TBA. The full scoring breakdowns on Match pages are a good example – if I wanted to send my friends who aren't in FIRST yet a match video to get them excited, the video is more important than the full breakdown. Providing these detailed stats in a separate space on the site might be a good way to accomplish both goals. For most people, the Event and Team pages will be much more streamlined and focused on the most common tasks of "What's happened and what's coming up?". For people who want to dive deeper, they can go to a dedicated Insights area to see more sophisticated stats breakdowns and analysis. Not as many people will go here as the main pages, but we can serve that audience much better with this standalone space. This would keep the core of the site focused on promoting awareness of FIRST and helping people watch Events, and creates a "insights stats portal" where we can geek out on providing the best analysis possible. |
|
@gregmarra put it better and clearer than my thoughts. Geeky details for super enthusiasts (like @Nuttyman54 -- Evan do you want to add your name to your GitHub profile?) are great as long as they don't interfere for most people. To hazard a guess: In a team of 100 people counting students, mentors, and family spectators, would maybe 2 - 5 be into detail stats? Maybe the FB group size provides a better estimate. Using predicted scores to pick matches to watch is clever. Maybe suggest that on its info page. Tools for scouting and "How are we doing?" are widely useful to teams and their followers, which is why I like #1465 . It'd be nice to not have to keep explaining, "Yes, but they played 1 (or 2) fewer matches than the other teams." |
|
I'm going to quickly throw something together that puts predictions on a separate page and see how it looks. This keeps the main event page cleaner and more accessible and less confusing to the general public. Maybe we should move the "Event Insights" tab (http://www.thebluealliance.com/event/2016micmp#event-insights) and OPR to this new page as well, and then we can really go all out with different statistics without worrying about cluttering up the main page. That way the main event page only has the bare minimum FACTS -- Match results, rankings (unaltered as provided by FIRST), awards, teams, district points, and general event info. |
|
I personally like the design @fangeugene posted above. I also agree with @gregmarra to put the switch next to the "Qualification Results" header. I think it does a good job of making it clear what matches have actually been played, and a good job of not inundating casual users with too much data. I would vote to get this feature out there as is, and to consider bigger organizational changes after Champs. |
Anybody got any good names to call this? |
|
This is linked to from the event page at The confidence is slightly handwavy. It assumes a variance in predicted score of 2500, which I chose to be large enough to be conservative.
|
|
That is pretty cool!
Show fewer significant digits? How much of the prediction error is due to penalty points? |
Good idea
Not sure. Are you just curious or do you suggest displaying it? |
|
Just curious, and betting that penalties are less predictable than scoring capabilities. |
|
@fangeugene you're on track to win the "Nate Silver" award for this year! |
|
Quick UI update Edit: We should probably bound negative scores. I'm not sure why that's wrong actually, it didn't used to do that. I'll check the math.
|
|
I'm going to land this tonight and not link to it anywhere. |
|
Hi All, Ether just pointed me to this conversation. It's great to see match predictions coming into TBA! I have included match predictions into my FTC2016 Android App this season. It uses OPRs to both predict the match scores as well as to predict the probability of a win. This second part is particularly useful as folks sometime don't know what to make of a predicted score being 40 points above or below that of the other alliance. Also, I encourage the stats folks to check out this paper and this CD thread discussing it. Of particular interest is the MMSE method of computing OPR compared to the traditional least squares (LS) method. The main advantages of the MMSE method are that it is stable even when only a few matches have been played (unlike the LS method) and it can naturally incorporate any apriori information for what the OPR is expected to be. The plots on page 38 in particular show how the MMSE OPRs are more stable early and more accurately predict the underlying offensive power compared to the LS OPRs. So, for example, if you think that the OPR for a team at an upcoming tournament might be 50 points with a standard deviation of 15 points (based on previous tournaments, or any other data you might have), the MMSE method naturally incorporates this knowledge into the running OPR estimates for the tournament. This gives a similar progressive change to the OPR estimates as a tournament happens, similar to the ixOPR, but without the adhoc iteration. It also works well when no apriori information is available (first tournament, etc.). |
|
I've added an Appendix D to the paper which explicitly shows how to compute MMSE OPR estimates with a-prior knowledge of the OPRs (from previous tournaments or scouting). |
|
Looks really interesting. I'll give it a more detailed read when I get the chance. |
Towards #1476
This could probably use a page that describes how predictions are calculated. Not sure how/where to link to it though. Aside from that, is it obvious enough to the user that these are predictions and they shouldn't sue us if they are wrong?😛