Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rounding of large numbers leading to incorrect results #5

Open
kvap opened this issue Feb 3, 2021 · 1 comment
Open

Rounding of large numbers leading to incorrect results #5

kvap opened this issue Feb 3, 2021 · 1 comment

Comments

@kvap
Copy link

kvap commented Feb 3, 2021

Enter 18446744073709549568 and you get the correct bit representation:
0 10000111110 1111111111111111111111111111111111111111111111111111.
But then it shows n = 18446744073709550000 and the input field also get updated to that value.

@bartaz
Copy link
Owner

bartaz commented Feb 3, 2021

Hey @kvap, thanks for trying it out!

It's been a while since I made it and worked with JS numbers in such detail, but as far as I remember this is expected behaviour.

The 32 bit format is limited to how large the numbers it can acurately represent are. In JS the largest integer that can be represented safely is Number.MAX_SAFE_INTEGER which is 9007199254740991 anything larger than that will be rounded to the closest value that can be represented.

So in case of number like 18446744073709549568, the closest possible value in 32-bit floating point is 18446744073709550000.
Even in browser JS console you can see that when you try to use such number, it shows the rounded value:
Screenshot 2021-02-03 at 13 00 28

So that's why, when you enter such number in my visualisation tool, it shows the 32 bit representation of the closest possible number, and updates the input to whatever given number is.

Hope it helps.

I believe that modern JS engines (node, modern browsers) may support other number formats, like BigInt that can be used to represent larger integers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants