Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweaks to hit statistics #4113

Open
jostephd opened this issue Jun 11, 2019 · 26 comments

Comments

Projects
None yet
5 participants
@jostephd
Copy link
Member

commented Jun 11, 2019

After #4070 added hit statistics I've received requests for changes.

  • Reverse the red-to-green coloring to be green-to-red, and vice versa (see 061edc2 for what's there now) - done 9e0000a
  • Change the tooltip to _"stats dialog^The first percentage is the probability of scoring fewer than 1745 hits. The second percentage is the probability of scoring more than 1745 hits." where 1745 is whatever the actual number of hits (shown in the adjacent column to the left) was. (edit: Removed "Quantiles" and "a priori". @GregoryLundberg suggested "central value" instead.) - superseded by 9e0000a
  • Maybe say "p value" somewhere (#1378) although that might be problematic for the same reason as "quantile" above
  • Add column headers to the percentages column, "Ratio" in the damage table and "Quantiles" (or some other word that more players would know) in the hits table. Alternatively, make the damage and hits table symmetrical in what they present
  • Consider removing the percentages - done 9e0000a
  • Split the "Overall" column in "Actual" and "Expected"
  • Add total number of strikes (not hits) - same as #4113 (comment)
  • Make hit stats similar to damage stats. Replace the (probability of hit less, probability of hit more) with (actual_hits / expected_hits - 1) * 100% as it is done for damage stats.
  • Place the (probability of hit less, probability of hit more) under the "detailed statistics" tab or spoiler.
  • Add some sort of histogram - see #4113 (comment)
  • Hide the "This turn" column header when viewing statistics of a previous scenario or of all scenarios - a4da90f
  • How about if instead of showing "93.4%, 6.1%" it would just show "94"? - done 9e0000a

@jostephd jostephd added the UI label Jun 11, 2019

@jostephd jostephd self-assigned this Jun 11, 2019

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

Added label "Ready for testing" because #4070 is already merged. If anyone else has changes to suggest please speak up now. Once all the feedback is in I'll see about implementing changes.

@soliton-

This comment has been minimized.

Copy link
Member

commented Jun 11, 2019

  • Split the "Overall" column in "Actual" and "Expected"
@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 11, 2019

WTF is "quantiles". Only reference I see to it with a Google search is a trademarked name for some sort of measurement system for mathematical education in primary schools.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

A generalization of the terms "median" (partition into two parts), "percentile" (partition into 100 parts), etc. https://en.wikipedia.org/wiki/Quantile

If there were 1745 hits out of 3000 strikes then we conceptually partition the distribution's PMF to two buckets: [0,1744] and [1746,3000].

If someone can suggest better wording I'll use it.

@slavrenyuk

This comment has been minimized.

Copy link

commented Jun 11, 2019

I will repeat my suggestion here. Make hit stats similar to damage stats. Replace the (probability of hit less, probability of hit more) with (actual_hits / expected_hits - 1) * 100% as it is done for damage stats.

Place the (probability of hit less, probability of hit more) under the "detailed statistics" tab or spoiler.

Split the "Overall" column in "Actual" and "Expected" is definitely an improvement.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

@slavrenyuk Added to the first post, and I'll repeat my countersuggestion in turn: make damage stats similar to hit stats by replacing (actual_hits / expected_hits - 1) * 100% with (probability of inflicting/taking less hitpoints, probability of inflicting/taking more hitpoints).

Why? Because when the dialog shows, say, 27 / 25.2 (+7%), the percentage is redundant for two reasons: because it can be derived from 27 and 25.2, and because at the end of a campaign it's almost always in the range of +2%/-2%, if some value is always about the same there is no need to show it in the UI.

That's also my answer to your first suggestion. The +2%/-2% thing is also true for the hits table. For example, in my WoV playthrough the hits table at the end of the campaign showed 1745/1703.2, that's a +2% difference, but the a priori probability of having so many hits was fifteen to one against.

@slavrenyuk

This comment has been minimized.

Copy link

commented Jun 11, 2019

@jostephd

when the dialog shows, say, 27 / 25.2 (+7%), the percentage is redundant for two reasons: because it can be derived from 27 and 25.2, and because at the end of a campaign it's almost always in the range of +2%/-2%, if some value is always about the same there is no need to show it in the UI.

  1. It can be derived, correct. Strictly speaking it is redundancy. From the user point of view it is friendly UI. Most of the users are not very interested in actual / expected damage, they are interested in the percentage value. Since it is recognized as a measure of luck.

  2. Actual and expected values on the long run (during the whole campaign) are expected to be similar. The most interesting is to see the per scenario stats.

@slavrenyuk

This comment has been minimized.

Copy link

commented Jun 11, 2019

Also I would recommend to carefully think about what Mawmoocn proposes. His ideas may look attractive since the guy is probably talented and skilled in math. However, he may end up proposing an AI that will enslave the humanity 😃 Think twice before implementing it.

@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2019

Well, that just shows Google isn't always your friend. As to what to replace "quantiles" with, I find plain English to be quite adequate and, most likely, far easier for the translators to deal with.

I suppose I should blush. I had an feeling I should know the term. Seeing the Wikipedia entry, I certainly knew it, 40 years ago. But, I guess that shows how obscure the term is that I could go a life-long career using applied mathematics and statistics and never needed it.


I sorta like the existing actual/expected (percentage) format. Yes, it's redundant. But it's sure convenient.

Sure, it can also be a tad bit misleading if people get too hung up on the percentage. "Wow, a 100% variance? You MUST be cheating!" (About a 2/1 +100% statistic.)

I would point out that 27/25.2 (+7%) is misleading in another way: you can't get 25.2. 25, yes, 26, yes, but not 25.2. So you're actually either 27/25(+8%) or 27/26(+4%).

So, when I care to look at the statistics, I tend to do some mental gymnastics to try to understand what they're really telling me, if anything at all.


Don't know what Mawmoocn suggestions are and not interested in digging for them on the Forums.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2019

Well, that just shows Google isn't always your friend. As to what to replace "quantiles" with, I find plain English to be quite adequate and, most likely, far easier for the translators to deal with.

Got any specific suggestions? I'd be happy to rephrase the tooltip or column headers, but if there's a plain English term for "a number that divides [0,100] into [0,x] and [x,100]", I must have forgotten it myself. :)

Question about the percentages that are displayed currently. Those of you who think the percentages should be removed, is that because they're unclear or because they're not something the statistics dialog should show (even assuming for the moment that we'd have clear documentation of what the percentages are and how to interpret them)?

Two current images for reference

These images are from the "All Scenarios" view of the statistics dialog, in both cases viewing the statistics of an entire campaign (AOI and WoV).

2019-06-11-114649_325x177_scrot
2019-06-11-112448_422x185_scrot

I'm asking because if the problem is lack of documentation, or lack of symmetry between the hits and damage tables, then maybe we can fix that problem but keep the percentages (quantiles) shown elsewhere. For example, we could replace the quantiles by actual/expected - 1 percentage, so the hits and damage tables would be symmetrical, and show the quantiles elsewhere, where it wouldn't be confusing, and add documentation explaining what those two percentages mean (the documentation wouldn't use the term "quantile"). I'm just trying not to throw the baby out with the bathwater here, not to remove the quantiles if the real problem is the expectation that the percentages are the ratio of actual hits to expected hits.

@slavrenyuk

This comment has been minimized.

Copy link

commented Jun 12, 2019

@jostephd

Those of you who think the percentages should be removed, is that because they're unclear or because they're not something the statistics dialog should show (even assuming for the moment that we'd have clear documentation of what the percentages are and how to interpret them)?

For most players (IMO):

  • The percentages would be unclear no matter how good and detailed documentation is provided. Also many of players will not even bother reading that documentation.
  • The percentages is not something that players are interested in.

For me:

  • Percentages are clear after I read the tooltip.
  • The percentages is not something that I am interested in.

Although I understand that there are different opinions. That's why I suggested to move the percentages under tab or spoiler called "detailed/additional statistics". Because majority of users are not interested in it (IMO). And it is still available for those who are interested.

@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2019

For a term, I'd try "central value" .. it's not totally correct but the meaning is plain and close enough.


Looking at your screen shots: where do you get TWO percentages from? Very unclear.

And why are they over under "This turn" .. well not really but sure looks like maybe it's just bad formatting. They must be for it, otherwise, why is it there and not used?

Going just by the screen shots, I'd suggest dropping a nuke on it all and starting over. Begin with writing it in English: if you can't clearly describe it in plain English, you certainly can't expect users to figure it out.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2019

Although I understand that there are different opinions. That's why I suggested to move the percentages under tab or spoiler called "detailed/additional statistics". Because majority of users are not interested in it (IMO). And it is still available for those who are interested.

Thanks. I'll wait for more opinions before I make any changes.

For a term, I'd try "central value" .. it's not totally correct but the meaning is plain and close enough.

Added to the first post.

Looking at your screen shots: where do you get TWO percentages from? Very unclear.

Did you read the tooltip? Can you suggest how to rephrase it?

And why are they over under "This turn" .. well not really but sure looks like maybe it's just bad formatting. They must be for it, otherwise, why is it there and not used?

It's there and not used because the screnshot is of the "All Scenarios" view, and in that view, no data is shown in the fourth and fifth columns. Those columns are used when viewing the statistics of the current scenario: #4070 (comment) I added to the first post a suggestion to remove the "This turn" column header in those cases. Note that this problem occurs in the damage table too, it's not new.

Going just by the screen shots, I'd suggest dropping a nuke on it all and starting over. Begin with writing it in English: if you can't clearly describe it in plain English, you certainly can't expect users to figure it out.

This does not even deserve a reply.

@slavrenyuk

This comment has been minimized.

Copy link

commented Jun 12, 2019

Thanks. I'll wait for more opinions before I make any changes.

Sounds good 👍

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

How about if instead of showing "93.4%, 6.1%" it would just show "94"? That is, just the percentile (without using that word)? Instead of showing 5,95 it would show 5. Instead of showing 95,5 it would show 95.

@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

Tooltip wasn't in your screen shot. I'm away from my Wesnoth-box so can't pull and check myself. A link into the gh tree will do if you don't have an image.

Truncating the fractional part instead of rounding to the nearest whole value sounds good. I was thinking we should round for psychology. Making the number appear smaller (less deviation from the expectation) should do that.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Tooltip wasn't in your screen shot. I'm away from my Wesnoth-box so can't pull and check myself. A link into the gh tree will do if you don't have an image.

Look at the first post of this thread, second bullet. The text there goes with the second screenshot in this comment.

Of course, if we end up showing just one number instead of two, as I suggested, then the tooltip will be rewritten anyway.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

It's there and not used because the screnshot is of the "All Scenarios" view, and in that view, no data is shown in the fourth and fifth columns. Those columns are used when viewing the statistics of the current scenario: #4070 (comment) I added to the first post a suggestion to remove the "This turn" column header in those cases. Note that this problem occurs in the damage table too, it's not new.

Fixed a4da90f

@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

Ah. The tool-tip is too much, as is having to values. The first question is why they don't total 100% in all cases. The second is do we really need to do B=100-A for the player? How about:

  • "The percentage shown is the probability of this score being the result from a large number of replays."
  • Only show the first percentage value.

The question I then have: is this really an expected-value calculation? That "a priori" bit makes me think the original author didn't really think so. "the probability of this score resulting from some large number of games, under conditions we can't describe, making assumptions we can't justify." might be more accurate. OK, I'm still feeling snarky.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

is this really an expected-value calculation

I think it is, but you're welcome to check my math. Look up the function tally added by #4070.

@Pentarctagon

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

As an aside, the PR that added this was previously open for a month - coming in now and saying the entire UI part of it should be nuked really is not constructive.

@GregoryLundberg

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

I know I'm being snarky.

Hmm. I think I see where my tredidation comes about rounding. Not sure it's worth the effort to do what I'd prefer (keep a running integer Hi and a running integer Low) value bracketing the value you're calculating as a real number. We could be more mathematically accurate but it would probably be even harder for players to understand.

@jostephd jostephd added this to the 1.15.0 milestone Jun 13, 2019

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

To those of you who proposed a histogram: I tried this on the "All Scenarios" view of my WoV playthrough where I took the 1745/93.4% screenshot. The data is:

30%: 27.7% (-2.3%)
40%: 45.1% (+5.1%)
50%: 53.7% (+3.7%)
60%: 60.5% (+0.5%)
70%: 71.0% (+1.0%)
80%: 81.5% (+1.5%)

That means: of all the strikes I attempted at 80% cth, 81.5% hit. I am leaning toward not adding this. Same for the overall number of strikes (1745 hits out of 3250 strikes).

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Not sure it's worth the effort to do what I'd prefer (keep a running integer Hi and a running integer Low) value bracketing the value you're calculating as a real number.

I don't understand what algorithm you just ruled out here.

@soliton-

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

I proposed a histogram with absolute strike values not percentages.

@jostephd

This comment has been minimized.

Copy link
Member Author

commented Jun 22, 2019

I'll change this again to Option C, though I might rearrange so instead of +19% (10 / 8.4) | 78.9 it'll be 10 / 8.4 | +19% | 78.9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.