Format initial string to specific precision, so rounding works. #4159

Merged
merged 1 commit into from Oct 12, 2016

Projects

None yet

2 participants

@headius
Member
headius commented Sep 15, 2016

The logic in our double-based sprintf logic attempts to round
values manually by inspecting a long-form version of the double.
If we need to round, and the next digit is a five, it will round
toward zero (truncate) iff that five is the last digit in the long
unrounded string. This appears to have been an attempt to mimic
C printf's behavior of rounding "true half" to even in the
presence of inaccurately-represented IEEE754 decimals.

This commit changes the pre-formatting to actually format with the
specified precision, allowing NumberFormat's default HALF_EVEN
logic to to the work for us.

Fixes #4157.

@headius headius Format initial string to specific precision, so rounding works.
The logic in our double-based sprintf logic attempts to round
values manually by inspecting a long-form version of the double.
If we need to round, and the next digit is a five, it will round
toward zero (truncate) iff that five is the last digit in the long
unrounded string. This appears to have been an attempt to mimic
C printf's behavior of rounding "true half" to even in the
presence of inaccurately-represented IEEE754 decimals.

This commit changes the pre-formatting to actually format with the
specified precision, allowing NumberFormat's default HALF_EVEN
logic to to the work for us.

Fixes #4157.
6cef061
@headius headius added this to the JRuby 9.1.6.0 milestone Sep 15, 2016
@enebo enebo merged commit 7f9866b into jruby:master Oct 12, 2016

0 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build failed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment