Calc: calculation error with decimal number #98

Closed
jrlacharnay opened this Issue Jul 23, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@jrlacharnay

Hello,

When entering the following calculation: "500.10-400", the displayed result is not "100.10" as expected, but "100.10000000000002".

My version:

10:16:14.390 Keypirinha 2.8 (246eefb) for x64
10:16:14.390 Portable mode.
10:16:14.390 System: WinNT-x64 10.0.10586-ws-0x0300

Thanks!
Jean-Rémy

@jrlacharnay

This comment has been minimized.

Show comment
Hide comment
@polyvertex

This comment has been minimized.

Show comment
Hide comment
@polyvertex

polyvertex Jul 23, 2016

Member

A way to deal with that would be for the Calc package to offer a "rounding" option so the user can specify the number of digits displayed after the separator by asking Calc to round it. This is something that can be added to a future release.

FYI and if that is of any interest to you, for now the displayed value is computed by the embedded Python interpreter and returned as-is by the Calc plugin. Python itself is dependent, at least partly, on the way the CPU deals with floating point arithmetic.

Member

polyvertex commented Jul 23, 2016

A way to deal with that would be for the Calc package to offer a "rounding" option so the user can specify the number of digits displayed after the separator by asking Calc to round it. This is something that can be added to a future release.

FYI and if that is of any interest to you, for now the displayed value is computed by the embedded Python interpreter and returned as-is by the Calc plugin. Python itself is dependent, at least partly, on the way the CPU deals with floating point arithmetic.

@ThaSiouL

This comment has been minimized.

Show comment
Hide comment
@ThaSiouL

ThaSiouL Aug 3, 2016

here is an updated calc version by myself that deals with the issue (among others)
Calc.zip
(Updated version from the version I posted in #70 and #93 )

EDIT by @polyvertex: @ThaSiouL has published its contribution in a GitHub repository

ThaSiouL commented Aug 3, 2016

here is an updated calc version by myself that deals with the issue (among others)
Calc.zip
(Updated version from the version I posted in #70 and #93 )

EDIT by @polyvertex: @ThaSiouL has published its contribution in a GitHub repository

This was referenced Aug 3, 2016

@ThaSiouL

This comment has been minimized.

Show comment
Hide comment
@ThaSiouL

ThaSiouL Aug 3, 2016

2016-08-03 20_18_52-program manager
including .ini decimal sign

ThaSiouL commented Aug 3, 2016

2016-08-03 20_18_52-program manager
including .ini decimal sign

@polyvertex

This comment has been minimized.

Show comment
Hide comment
@polyvertex

polyvertex Aug 20, 2016

Member

The Calc package has been greatly improved and released with v2.9.

Member

polyvertex commented Aug 20, 2016

The Calc package has been greatly improved and released with v2.9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment