Precalculated instance stats are scaled incorrectly

[bhollis]

I've been working on switching DIM from using the precalculated stats to calculating them myself using StatGroups and displayInterpolation. In doing so, I noticed a bug where the precalculated instance stats from the API do not agree with properly interpolated stats (and the properly interpolated stats DO agree with the game!)

It looks like maybe the API is doing this to interpolate the value:

1. Find start, the entry in displayInterpolation whose value is just below the raw stat value.
2. Find end, the entry in displayInterpolation whose value is just above the raw stat value.`
3. Calculate a scaling factor t as t = (value - start.value) / end.value
4. Calculate the final stat value as Math.round(start.weight + t * (end.weight - start.weight)).

Step 3 is incorrect - the denominator should be the range between start and end, not just end. It works "most of the time" because start.value is very often 0.

The correct formula for t is t = (value - start.value) / (end.value - start.value), which produces the same value as in-game for all stats.

[vthornheart-bng]

I spent some time examining this - indeed, you hit the nail on the head about how we do interpolation for most stats! Though when I examined our algorithm, it matches the one you describe here, and I was able to verify that on all of the stats (except for Magazine Size) on several items across multiple characters. There are some interesting details in rounding that we use vs. other potential algorithms (we use round to nearest even/"Banker's rounding") which we are perhaps mismatching on and thus causing the apparent discrepancy between you and I. But as far as the game discrepancy vs. the API, that's another more complicated story, and should only be happening for a single stat (Magazine Size).

I mention all of this to make sure folks know about the limitations of our stats system. I spoke with bhollis offline, and the stat having trouble in particular is the Magazine Size stat. Unfortunately, this specific stat is calculated in an entirely different way from all other stats, and it uses algorithms and scripts that Bungie.net doesn't have access to and cannot process. Unfortunately, as a result we will never be able to provide accurate magazine size stats.

I have a theory at the moment, which is that if our rounding algorithms are different, the scripts that the game server executes for magazine size may be using the same rounding algorithm that you are, whereas the other stats are using the banker's algorithm I mentioned above: and that when the stat happens to get the right value for you, it's because there aren't any further modifications happening on the magazine size stat at the time when you're inspecting it in-game and your code happens to round the right way for it. However, I will need to talk with folks who have an understanding of that codebase to find out for sure.

Anyways, the disclaimer I need to provide to folks is that you should never trust the Magazine Size stat in Bungie.Net as canon. It is provided only for historical/comparative reasons. We were going to take it out late in D1, but left it in because folks found it useful for the sake of comparative analysis. We will never be able to provide a truly accurate number for magazine size.

[floatingatoll]

Thank you for the disclaimer. Given that disclaimer, I would still say that rounding repairs to this otherwise-inaccurate stat would still be nice-to-have simply because it can be frustrating trying to separate _accidentally_ inaccuracy from _intentional_ inaccuracy.

…

On Sep 4, 2019, at 14:50, Vendal Thornheart ***@***.***> wrote: I spent some time examining this - indeed, you hit the nail on the head about how we do interpolation for most stats! Though when I examined our algorithm, it matches the one you describe here, and I was able to verify that on all of the stats (except for Magazine Size) on several items across multiple characters. There are some interesting details in rounding that we use vs. other potential algorithms (we use round to nearest even/"Banker's rounding") which we are perhaps mismatching on and thus causing the apparent discrepancy between you and I. But as far as the game discrepancy vs. the API, that's another more complicated story, and should only be happening for a single stat (Magazine Size). I mention all of this to make sure folks know about the limitations of our stats system. I spoke with bhollis offline, and the stat having trouble in particular is the Magazine Size stat. Unfortunately, this specific stat is calculated in an entirely different way from all other stats, and it uses algorithms and scripts that Bungie.net doesn't have access to and cannot process. Unfortunately, as a result we will never be able to provide accurate magazine size stats. I have a theory at the moment, which is that if our rounding algorithms are different, the scripts that the game server executes for magazine size may be using the same rounding algorithm that you are, whereas the other stats are using the banker's algorithm I mentioned above: and that when the stat happens to get the right value for you, it's because there aren't any further modifications happening on the magazine size stat at the time when you're inspecting it in-game and your code happens to round the right way for it. However, I will need to talk with folks who have an understanding of that codebase to find out for sure. Anyways, the disclaimer I need to provide to folks is that you should never trust the Magazine Size stat in Bungie.Net as canon. It is provided only for historical/comparative reasons. We were going to take it out late in D1, but left it in because folks found it useful for the sake of comparative analysis. We will never be able to provide a truly accurate number for magazine size. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1029?email_source=notifications&email_token=AAAWUDBMEFZE4EGKPQVZGUDQIAUSXA5CNFSM4ITUGRIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD55DNIQ#issuecomment-528103074>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAWUDH445R2IV623DG4MIDQIAUSXANCNFSM4ITUGRIA>.

[vthornheart-bng]

I can't guarantee that the problem will actually be globally resolved by changing rounding for that stat: right now it's a theory that I have about the specific items we're looking at in the specific state in which we're looking at them (while in orbit). Before I make a change I need to verify if my theory is accurate. It's very possible that the specific examples we're looking at just happen to align well with the theory, but that it won't hold for the general use case. If it is what's happening, I'll gladly change it for Magazine Size. It still won't be accurate outside of that specific scenario (i.e. items being examined while in orbit and not equipped), but at least it would be more right more often!

[bhollis]

Just as a clarification - when you say you use Banker's Rounding, is that for all stats, or only Magazine Size?

[EdgarVerona]

Banker's rounding is used for every stat, aside from my speculation (which I have not had time to investigate yet) around the possibility of magazine size behaving differently.

[bhollis]

Thanks! I've verified that using banker's rounding helps us more closely match the API's precalculated stats.