Statistical accuracy PP and difficulty scaling for the osu!taiko ruleset #20963

Natelytle · 2022-10-27T04:00:58Z

Huge thanks to Frost for the majority of the math behind this rework, and to LTCA for helping me balance it.

Estimates UR from the play, and scales accuracy with it

Changes

Adds deviation estimation to the osu!taiko ruleset
Replaces the old accuracy formulas with new ones fit for the estimated UR values
Also includes some balancing changes LTCA decided would be best

Reasoning

The old accuracy PP formula did not scale well with higher overall difficulties, punishing lower accuracy more than it rewarded higher OD.
SR scaling was not affected by OD at all.
Unstable rate is an easier metric to work with, as there is a true "perfect" value.

Estimation Theory

In order to estimate UR, we assume all hits are normally distributed, with a mean of ±0 and a deviation σ. This gives us the probability that with a certain σ, any given hit gives a certain hit result (300, 100, miss). We can compare these percentages to the true percentages of any given judgement in a play, and return whichever σ value is the closest match.

Further documentation can be found in a google doc here.

This estimation requires MathNet.Numerics, a package for advanced mathematical formulas not present in C#.

SR/PP sheets:

Converts:
No converts:
Converts, ranked-only:
No converts, ranked-only: https://docs.google.com/spreadsheets/d/1M4FppnFUvf5YRsPRmNwC81XnQ4qKOPVmAwkWeBCq-F8/edit

As of 3a609c9

osu.Game.Rulesets.Taiko/osu.Game.Rulesets.Taiko.csproj

Lawtrohux · 2022-10-27T06:46:35Z

@smoogipoo can we get two smoogisheet/s for this and #20558 combined with converts enabled/disabled?

Lawtrohux

Overall, really like the direction of the PR and the initial results I saw while testing and balancing.

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceCalculator.cs

vunyunt

Aside from the question I had about buffing hard rock specifically, everything else looks good here.

One thing to note relating to the WIP rhythm rework is that it may take hit windows into consideration in a way that's not just a multiplier, we might be able to borrow some ideas from here. But we also want to be careful as to not twice-consider the effect of hit windows, as to avoid making high OD (high rhythm) map overrated.

vunyunt · 2022-10-30T09:50:25Z

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceCalculator.cs

@@ -84,35 +86,52 @@ private double computeDifficultyValue(ScoreInfo score, TaikoDifficultyAttributes
                difficultyValue *= 1.025;

            if (score.Mods.Any(m => m is ModHardRock))
-                difficultyValue *= 1.050;
+                difficultyValue *= 1.10;


Since the changes here already account for hit windows properly, and does not concern SV, why does hard rock specifically need to be buffed here?

This was done in balancing, as mid-range accuracy with HR was pretty underweighted (4x 100 on the limit does not exist HR was worth 20pp less than a HD SS). Though @Natelytle could the model be fit for greater accuracy leniency on super high OD's?

I don't think you can without increasing complexity and decreasing estimation accuracy, I think a HR multiplier buff is a better direction if HR in particular is underweight

Natelytle · 2022-10-30T19:03:07Z

Low end was gaining too much (doubling in some cases) and LTCA said it would be good if I could buff high end a bit so I made the SR multiplier harsher at the low end and a bit more of a buff to the high end (feedback on that would be appreciated once smoogisheet is out)

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceAttributes.cs

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceCalculator.cs

vunyunt · 2022-11-14T04:59:31Z

@smoogipoo Requesting a smoogisheet for review, thanks in advance!

smoogipoo · 2022-11-14T07:28:25Z

Will need conflict resolution on this PR, since the HDFL multiplier changed in both this and the previous PR.

smoogipoo · 2022-11-14T07:32:47Z

Will generate sheet with 1.1x HDFL multiplier until a decision has been made/this branchs' merge conflict is resolved.

Lawtrohux · 2022-11-14T07:44:39Z

Either will be fine, however if possible this PR's would be ideal to offset the notion of OD being better represented. I also believe that 1.05x is a better either way.

Lawtrohux

This looks good to me now. While there are some values that are higher than ideal, this is intrinsically due to other parts of both the SR and PP calculations, not to statistical accuracy. Having this as the 'clean branch' will be ideal.

Natelytle · 2024-03-10T05:39:56Z

This should be close to merge ready now, just one more sheet maybe. No external libraries are required anymore and switching to a confidence interval based system solved the issue of some values being too high for the devs' liking.

smoogipoo · 2024-03-11T09:45:29Z

!diffcalc
RULESET=taiko

github-actions · 2024-03-11T09:45:49Z

Target: #20963
Spreadsheet: https://docs.google.com/spreadsheets/d/1L8b-BiV6qyLnHe_gGZWti8Rn3AWksL6pC_CA8KYZVts/edit

Lawtrohux

After reviewing profile based values, and the smoogisheet, this LGTM. While there are still problem maps, that is a reflection of the colour system rather than statistical accuracy.

I'm unsure of utisiling utils as a split off from mathNET. However its code is just localising so I don't see an issue.

stanriders · 2024-05-27T16:37:24Z

@smoogipoo this PR is considered approved by taiko pp committee and is ready for merge

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceAttributes.cs

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceCalculator.cs

bdach · 2024-05-29T06:27:27Z

Deployment considerations:

1 added difficulty attribute, which comes out to
- 13 bytes per row
- taiko has 4 DifficultyAdjustmentMods which is $2^4 = 16$ possible combinations
- over ~26000 taiko-specific beatmaps + ~111700 converts
- totalling ~28 MB of extra storage (not significant)
- plus the need to do a full run of taiko diffcalc server-side

Other than that, I see no further roadblocks. Would like to see the review above addressed though (especially regarding the possible crash).

bdach · 2024-10-07T12:18:47Z

@Natelytle please check merge conflict resolution when able (especially the difficulty attribute numbering - no idea why that attribute was assigned 29 in the first place).

bdach · 2024-10-07T13:29:17Z

!diffcalc
RULESET=taiko

github-actions · 2024-10-08T07:43:47Z

Target: #20963
Spreadsheet: https://docs.google.com/spreadsheets/d/1Sd51zO_t6JEyzRAZhvA36vjMN1PVh2EQgldy4OFDjHM/edit

Natelytle added 6 commits October 25, 2022 17:41

Implement taiko deviation estimation

442e68a

Fix difficultyvalue acc scaling

d5b06ae

oops

607a006

Slight adjustments

87cba2d

LTCA Balancing pass

7d3338a

harshen deviation scaling

af919a6

pull-request-size bot added the size/M label Oct 27, 2022

Fix formatting

2940d18

Lawtrohux added ruleset/osu!taiko area:difficulty labels Oct 27, 2022

peppy self-requested a review October 27, 2022 04:23

peppy reviewed Oct 27, 2022

View reviewed changes

osu.Game.Rulesets.Taiko/osu.Game.Rulesets.Taiko.csproj Outdated Show resolved Hide resolved

bdach requested a review from a team October 27, 2022 19:07

Lawtrohux suggested changes Oct 28, 2022

View reviewed changes

Natelytle added 3 commits October 28, 2022 16:18

Return null instead of infinity

883790c

remove other infinity reference

01c79d8

Return null for greatprobability >= 1

7403c1c

Lawtrohux requested a review from vunyunt October 29, 2022 03:28

vunyunt reviewed Oct 30, 2022

View reviewed changes

Fix low end accuracy, buff high end

16301f0

Lawtrohux suggested changes Nov 1, 2022

View reviewed changes

fix formatting

37c21cd

Natelytle added 2 commits November 24, 2022 19:09

account for low acc FC deviation

2ba1634

totalvalue

0e4e92b

Reduce accuracy scaling

5f0020b

Lawtrohux requested a review from a team July 31, 2023 00:41

smoogipoo mentioned this pull request Jul 31, 2023

Convert legacy total score to standardised when importing high scores ppy/osu-queue-score-statistics#135

Merged

2 tasks

Lawtrohux approved these changes Aug 4, 2023

View reviewed changes

vunyunt approved these changes Aug 5, 2023

View reviewed changes

Merge branch 'master' into taikostatacc

3a609c9

NiceAesth mentioned this pull request Aug 9, 2023

Aim doubletap detection #24488

Closed

Natelytle added 5 commits March 9, 2024 22:33

Merge master

8a26cda

Compute the upper bound on deviation with a 99% confidence interval

caba051

Include misses in the great window deviation calc

6ddb2b7

Fix comment

5370595

Remove MathNet.Numerics dependency

a9b3416

pull-request-size bot added size/XL and removed size/M labels Mar 10, 2024

Lawtrohux approved these changes Mar 18, 2024

View reviewed changes

bdach reviewed May 29, 2024

View reviewed changes

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceAttributes.cs Outdated Show resolved Hide resolved

osu.Game.Rulesets.Taiko/Difficulty/TaikoPerformanceCalculator.cs Outdated Show resolved Hide resolved

Natelytle added 3 commits May 29, 2024 09:40

Save deviation calculations to variables

1714567

Fix naming convention

f8f18b6

Move the return value for deviation below the local functions

2fb22f1

bdach added the next release Pull requests which are almost there. We'll aim to get them in the next release, but no guarantees! label Oct 7, 2024

Merge branch 'master' into taikostatacc

84d6467

bdach merged commit 6608d05 into ppy:master Oct 7, 2024
10 of 13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Statistical accuracy PP and difficulty scaling for the osu!taiko ruleset #20963

Statistical accuracy PP and difficulty scaling for the osu!taiko ruleset #20963

Natelytle commented Oct 27, 2022 •

edited by smoogipoo

Loading

Lawtrohux commented Oct 27, 2022

Lawtrohux left a comment

vunyunt left a comment

vunyunt Oct 30, 2022

Lawtrohux Oct 30, 2022 •

edited

Loading

Natelytle Oct 30, 2022 •

edited

Loading

Natelytle commented Oct 30, 2022 •

edited

Loading

vunyunt commented Nov 14, 2022

smoogipoo commented Nov 14, 2022

smoogipoo commented Nov 14, 2022

Lawtrohux commented Nov 14, 2022 •

edited

Loading

Lawtrohux left a comment

Natelytle commented Mar 10, 2024

smoogipoo commented Mar 11, 2024

github-actions bot commented Mar 11, 2024 •

edited

Loading

Lawtrohux left a comment

stanriders commented May 27, 2024

bdach commented May 29, 2024

bdach commented Oct 7, 2024 •

edited

Loading

bdach commented Oct 7, 2024

github-actions bot commented Oct 8, 2024

Statistical accuracy PP and difficulty scaling for the osu!taiko ruleset #20963

Statistical accuracy PP and difficulty scaling for the osu!taiko ruleset #20963

Conversation

Natelytle commented Oct 27, 2022 • edited by smoogipoo Loading

Estimates UR from the play, and scales accuracy with it

Changes

Reasoning

Estimation Theory

Lawtrohux commented Oct 27, 2022

Lawtrohux left a comment

Choose a reason for hiding this comment

vunyunt left a comment

Choose a reason for hiding this comment

vunyunt Oct 30, 2022

Choose a reason for hiding this comment

Lawtrohux Oct 30, 2022 • edited Loading

Choose a reason for hiding this comment

Natelytle Oct 30, 2022 • edited Loading

Choose a reason for hiding this comment

Natelytle commented Oct 30, 2022 • edited Loading

vunyunt commented Nov 14, 2022

smoogipoo commented Nov 14, 2022

smoogipoo commented Nov 14, 2022

Lawtrohux commented Nov 14, 2022 • edited Loading

Lawtrohux left a comment

Choose a reason for hiding this comment

Natelytle commented Mar 10, 2024

smoogipoo commented Mar 11, 2024

github-actions bot commented Mar 11, 2024 • edited Loading

Lawtrohux left a comment

Choose a reason for hiding this comment

stanriders commented May 27, 2024

bdach commented May 29, 2024

bdach commented Oct 7, 2024 • edited Loading

bdach commented Oct 7, 2024

github-actions bot commented Oct 8, 2024

Natelytle commented Oct 27, 2022 •

edited by smoogipoo

Loading

Lawtrohux Oct 30, 2022 •

edited

Loading

Natelytle Oct 30, 2022 •

edited

Loading

Natelytle commented Oct 30, 2022 •

edited

Loading

Lawtrohux commented Nov 14, 2022 •

edited

Loading

github-actions bot commented Mar 11, 2024 •

edited

Loading

bdach commented Oct 7, 2024 •

edited

Loading