Tweaks to hit statistics #4113

Open
opened this issue Jun 11, 2019 · 26 comments

Projects
None yet
5 participants
Member

jostephd commented Jun 11, 2019 • edited

 After #4070 added hit statistics I've received requests for changes. Reverse the red-to-green coloring to be green-to-red, and vice versa (see 061edc2 for what's there now) - done 9e0000a Change the tooltip to `_"stats dialog^The first percentage is the probability of scoring fewer than 1745 hits. The second percentage is the probability of scoring more than 1745 hits."` where 1745 is whatever the actual number of hits (shown in the adjacent column to the left) was. (edit: Removed "Quantiles" and "a priori". @GregoryLundberg suggested "central value" instead.) - superseded by 9e0000a Maybe say "p value" somewhere (#1378) although that might be problematic for the same reason as "quantile" above Add column headers to the percentages column, "Ratio" in the damage table and "Quantiles" (or some other word that more players would know) in the hits table. Alternatively, make the damage and hits table symmetrical in what they present Consider removing the percentages - done 9e0000a Split the "Overall" column in "Actual" and "Expected" Add total number of strikes (not hits) - same as #4113 (comment) Make hit stats similar to damage stats. Replace the (probability of hit less, probability of hit more) with (actual_hits / expected_hits - 1) * 100% as it is done for damage stats. Place the (probability of hit less, probability of hit more) under the "detailed statistics" tab or spoiler. Add some sort of histogram - see #4113 (comment) Hide the "This turn" column header when viewing statistics of a previous scenario or of all scenarios - a4da90f How about if instead of showing "93.4%, 6.1%" it would just show "94"? - done 9e0000a

Member Author

jostephd commented Jun 11, 2019

 Added label "Ready for testing" because #4070 is already merged. If anyone else has changes to suggest please speak up now. Once all the feedback is in I'll see about implementing changes.
Member

soliton- commented Jun 11, 2019 • edited by jostephd

 Split the "Overall" column in "Actual" and "Expected"
Contributor

GregoryLundberg commented Jun 11, 2019

 WTF is "quantiles". Only reference I see to it with a Google search is a trademarked name for some sort of measurement system for mathematical education in primary schools.
Member Author

jostephd commented Jun 11, 2019

 A generalization of the terms "median" (partition into two parts), "percentile" (partition into 100 parts), etc. https://en.wikipedia.org/wiki/Quantile If there were 1745 hits out of 3000 strikes then we conceptually partition the distribution's PMF to two buckets: [0,1744] and [1746,3000]. If someone can suggest better wording I'll use it.

slavrenyuk commented Jun 11, 2019

 I will repeat my suggestion here. Make hit stats similar to damage stats. Replace the (probability of hit less, probability of hit more) with (actual_hits / expected_hits - 1) * 100% as it is done for damage stats. Place the (probability of hit less, probability of hit more) under the "detailed statistics" tab or spoiler. Split the "Overall" column in "Actual" and "Expected" is definitely an improvement.
Member Author

jostephd commented Jun 11, 2019

 @slavrenyuk Added to the first post, and I'll repeat my countersuggestion in turn: make damage stats similar to hit stats by replacing (actual_hits / expected_hits - 1) * 100% with (probability of inflicting/taking less hitpoints, probability of inflicting/taking more hitpoints). Why? Because when the dialog shows, say, 27 / 25.2 (+7%), the percentage is redundant for two reasons: because it can be derived from 27 and 25.2, and because at the end of a campaign it's almost always in the range of +2%/-2%, if some value is always about the same there is no need to show it in the UI. That's also my answer to your first suggestion. The +2%/-2% thing is also true for the hits table. For example, in my WoV playthrough the hits table at the end of the campaign showed 1745/1703.2, that's a +2% difference, but the a priori probability of having so many hits was fifteen to one against.

slavrenyuk commented Jun 11, 2019

 @jostephd when the dialog shows, say, 27 / 25.2 (+7%), the percentage is redundant for two reasons: because it can be derived from 27 and 25.2, and because at the end of a campaign it's almost always in the range of +2%/-2%, if some value is always about the same there is no need to show it in the UI. It can be derived, correct. Strictly speaking it is redundancy. From the user point of view it is friendly UI. Most of the users are not very interested in actual / expected damage, they are interested in the percentage value. Since it is recognized as a measure of luck. Actual and expected values on the long run (during the whole campaign) are expected to be similar. The most interesting is to see the per scenario stats.

slavrenyuk commented Jun 11, 2019

 Also I would recommend to carefully think about what Mawmoocn proposes. His ideas may look attractive since the guy is probably talented and skilled in math. However, he may end up proposing an AI that will enslave the humanity 😃 Think twice before implementing it.
Contributor

GregoryLundberg commented Jun 12, 2019

 Well, that just shows Google isn't always your friend. As to what to replace "quantiles" with, I find plain English to be quite adequate and, most likely, far easier for the translators to deal with. I suppose I should blush. I had an feeling I should know the term. Seeing the Wikipedia entry, I certainly knew it, 40 years ago. But, I guess that shows how obscure the term is that I could go a life-long career using applied mathematics and statistics and never needed it. I sorta like the existing actual/expected (percentage) format. Yes, it's redundant. But it's sure convenient. Sure, it can also be a tad bit misleading if people get too hung up on the percentage. "Wow, a 100% variance? You MUST be cheating!" (About a 2/1 +100% statistic.) I would point out that 27/25.2 (+7%) is misleading in another way: you can't get 25.2. 25, yes, 26, yes, but not 25.2. So you're actually either 27/25(+8%) or 27/26(+4%). So, when I care to look at the statistics, I tend to do some mental gymnastics to try to understand what they're really telling me, if anything at all. Don't know what Mawmoocn suggestions are and not interested in digging for them on the Forums.
Member Author

jostephd commented Jun 12, 2019

 Well, that just shows Google isn't always your friend. As to what to replace "quantiles" with, I find plain English to be quite adequate and, most likely, far easier for the translators to deal with. Got any specific suggestions? I'd be happy to rephrase the tooltip or column headers, but if there's a plain English term for "a number that divides [0,100] into [0,x] and [x,100]", I must have forgotten it myself. :) Question about the percentages that are displayed currently. Those of you who think the percentages should be removed, is that because they're unclear or because they're not something the statistics dialog should show (even assuming for the moment that we'd have clear documentation of what the percentages are and how to interpret them)? Two current images for reference These images are from the "All Scenarios" view of the statistics dialog, in both cases viewing the statistics of an entire campaign (AOI and WoV). I'm asking because if the problem is lack of documentation, or lack of symmetry between the hits and damage tables, then maybe we can fix that problem but keep the percentages (quantiles) shown elsewhere. For example, we could replace the quantiles by `actual/expected - 1` percentage, so the hits and damage tables would be symmetrical, and show the quantiles elsewhere, where it wouldn't be confusing, and add documentation explaining what those two percentages mean (the documentation wouldn't use the term "quantile"). I'm just trying not to throw the baby out with the bathwater here, not to remove the quantiles if the real problem is the expectation that the percentages are the ratio of actual hits to expected hits.

slavrenyuk commented Jun 12, 2019

 @jostephd Those of you who think the percentages should be removed, is that because they're unclear or because they're not something the statistics dialog should show (even assuming for the moment that we'd have clear documentation of what the percentages are and how to interpret them)? For most players (IMO): The percentages would be unclear no matter how good and detailed documentation is provided. Also many of players will not even bother reading that documentation. The percentages is not something that players are interested in. For me: Percentages are clear after I read the tooltip. The percentages is not something that I am interested in. Although I understand that there are different opinions. That's why I suggested to move the percentages under tab or spoiler called "detailed/additional statistics". Because majority of users are not interested in it (IMO). And it is still available for those who are interested.
Contributor

GregoryLundberg commented Jun 12, 2019

 For a term, I'd try "central value" .. it's not totally correct but the meaning is plain and close enough. Looking at your screen shots: where do you get TWO percentages from? Very unclear. And why are they over under "This turn" .. well not really but sure looks like maybe it's just bad formatting. They must be for it, otherwise, why is it there and not used? Going just by the screen shots, I'd suggest dropping a nuke on it all and starting over. Begin with writing it in English: if you can't clearly describe it in plain English, you certainly can't expect users to figure it out.
Member Author

jostephd commented Jun 12, 2019

 Although I understand that there are different opinions. That's why I suggested to move the percentages under tab or spoiler called "detailed/additional statistics". Because majority of users are not interested in it (IMO). And it is still available for those who are interested. Thanks. I'll wait for more opinions before I make any changes. For a term, I'd try "central value" .. it's not totally correct but the meaning is plain and close enough. Added to the first post. Looking at your screen shots: where do you get TWO percentages from? Very unclear. Did you read the tooltip? Can you suggest how to rephrase it? And why are they over under "This turn" .. well not really but sure looks like maybe it's just bad formatting. They must be for it, otherwise, why is it there and not used? It's there and not used because the screnshot is of the "All Scenarios" view, and in that view, no data is shown in the fourth and fifth columns. Those columns are used when viewing the statistics of the current scenario: #4070 (comment) I added to the first post a suggestion to remove the "This turn" column header in those cases. Note that this problem occurs in the damage table too, it's not new. Going just by the screen shots, I'd suggest dropping a nuke on it all and starting over. Begin with writing it in English: if you can't clearly describe it in plain English, you certainly can't expect users to figure it out. This does not even deserve a reply.

slavrenyuk commented Jun 12, 2019

 Thanks. I'll wait for more opinions before I make any changes. Sounds good 👍
Member Author

jostephd commented Jun 13, 2019 • edited

 How about if instead of showing "93.4%, 6.1%" it would just show "94"? That is, just the percentile (without using that word)? Instead of showing 5,95 it would show 5. Instead of showing 95,5 it would show 95.
Contributor

GregoryLundberg commented Jun 13, 2019

 Tooltip wasn't in your screen shot. I'm away from my Wesnoth-box so can't pull and check myself. A link into the gh tree will do if you don't have an image. Truncating the fractional part instead of rounding to the nearest whole value sounds good. I was thinking we should round for psychology. Making the number appear smaller (less deviation from the expectation) should do that.
Member Author

jostephd commented Jun 13, 2019

 Tooltip wasn't in your screen shot. I'm away from my Wesnoth-box so can't pull and check myself. A link into the gh tree will do if you don't have an image. Look at the first post of this thread, second bullet. The text there goes with the second screenshot in this comment. Of course, if we end up showing just one number instead of two, as I suggested, then the tooltip will be rewritten anyway.
Member Author

jostephd commented Jun 13, 2019

 It's there and not used because the screnshot is of the "All Scenarios" view, and in that view, no data is shown in the fourth and fifth columns. Those columns are used when viewing the statistics of the current scenario: #4070 (comment) I added to the first post a suggestion to remove the "This turn" column header in those cases. Note that this problem occurs in the damage table too, it's not new. Fixed a4da90f
Contributor

GregoryLundberg commented Jun 13, 2019 • edited

 Ah. The tool-tip is too much, as is having to values. The first question is why they don't total 100% in all cases. The second is do we really need to do B=100-A for the player? How about: "The percentage shown is the probability of this score being the result from a large number of replays." Only show the first percentage value. The question I then have: is this really an expected-value calculation? That "a priori" bit makes me think the original author didn't really think so. "the probability of this score resulting from some large number of games, under conditions we can't describe, making assumptions we can't justify." might be more accurate. OK, I'm still feeling snarky.
Member Author

jostephd commented Jun 13, 2019

 is this really an expected-value calculation I think it is, but you're welcome to check my math. Look up the function `tally` added by #4070.
Member

Pentarctagon commented Jun 13, 2019

 As an aside, the PR that added this was previously open for a month - coming in now and saying the entire UI part of it should be nuked really is not constructive.
Contributor

GregoryLundberg commented Jun 13, 2019

 I know I'm being snarky. Hmm. I think I see where my tredidation comes about rounding. Not sure it's worth the effort to do what I'd prefer (keep a running integer Hi and a running integer Low) value bracketing the value you're calculating as a real number. We could be more mathematically accurate but it would probably be even harder for players to understand.

Member Author

jostephd commented Jun 13, 2019 • edited

 To those of you who proposed a histogram: I tried this on the "All Scenarios" view of my WoV playthrough where I took the 1745/93.4% screenshot. The data is: ``````30%: 27.7% (-2.3%) 40%: 45.1% (+5.1%) 50%: 53.7% (+3.7%) 60%: 60.5% (+0.5%) 70%: 71.0% (+1.0%) 80%: 81.5% (+1.5%) `````` That means: of all the strikes I attempted at 80% cth, 81.5% hit. I am leaning toward not adding this. Same for the overall number of strikes (1745 hits out of 3250 strikes).
Member Author

jostephd commented Jun 13, 2019

 Not sure it's worth the effort to do what I'd prefer (keep a running integer Hi and a running integer Low) value bracketing the value you're calculating as a real number. I don't understand what algorithm you just ruled out here.
Member

soliton- commented Jun 13, 2019

 I proposed a histogram with absolute strike values not percentages.

Open

Member Author

jostephd commented Jun 22, 2019

 I'll change this again to Option C, though I might rearrange so instead of `+19% (10 / 8.4) | 78.9` it'll be `10 / 8.4 | +19% | 78.9`.