Floating point error in Arithmetic command, given inputs of mixed precision

[yegeniy]

Description

77350.77+30219.12+91520.32+75374.84+3390 should result in 277855.05, not 277855.05000000005

Steps to reproduce

1. Type 77350.77+30219.12+91520.32+75374.84+3390
2. Run "Arithmetic" via the command palette
3. Observe the discrepancy between the expected response and the provided one

Expected behavior

I expected the result to have been 277855.05

Actual behavior

277855.05000000005 is provided instead.

Environment

• Build: 3176
• macOS 10.14.2

[yegeniy]

Note that when there are two decimal points, it seems to work as expected:

54.37 + 6479.72 + 3579.57 = 10113.66, as expected

[FichteFoll]

This is a result of how ploating point arithmetic works using the IEEE 754 standard. While Python can handle arbitrarily large whole numbers, floating point precision is limited when performed using standard operations.

Related: SublimeText/InsertNums#17 (comment)

[yegeniy]

Excellent write-up and discussion there, @FichteFoll.

[marcospb19]

Note that when there are two decimal points, it seems to work as expected:

That's not true, it depends on the numbers you're using, mess around a bit with different combinations and you can observe that any slight difference on any number can trigger or untrigger this behavior.

Python even contains the function math.fsum(iter) for trying to sum floating integers without losing any precision, but sometimes (e.g. your test case) it's not able to avoid it.

[marcospb19]

That said, we can use from decimal import Decimal to treat the numbers, but this would require some string processing before the eval() function call.

I could implement this. But I'm not sure if the official maintainers want to add complexity to this code, so I'll just wait for any heads up from them here. :D

[BenjaminSchaaf]

Note Decimal also has limited accuracy (albeit more than hardware floating point) and also wouldn't help in this situation without doing our own parsing which prevent/complicate using other python with the arithmetic command. I don't think this issue is actionable.