Kernel.sprintf("%f",...) rounds to even on MRI but JRuby rounds away from zero

[felixvf]

# for ruby_implementation in jruby-9.1.5.0 ruby-2.3.1; do rvm "$ruby_implementation" do ruby -e 'require "bigdecimal"; puts("#{"%5s" % RUBY_ENGINE}: #{"%3.1f" % BigDecimal.new("-1.2499999999999999")}")'; done
jruby: -1.3
 ruby: -1.2

Environment

JRuby 9.1.5.0 on 1.8.0_101 on x86-64

Expected Behavior

"%3.1f" % BigDecimal.new("-1.2499999999999999") should return "-1.2".

Actual Behavior

"%3.1f" % BigDecimal.new("-1.2499999999999999") actually returns "-1.3".

Additional Notes

The primary problem is consistency across Ruby implementations.
It would be also ok that "%3.1f" % BigDecimal.new("-1.2499999999999999") actually returned "-1.3" if MRI would also return the same value, which it currently does not.

One additional problem is that, apparently, a BigDecimal is first converted into a Float (and thereby looses precision) during the "%f" formatting. This is surprising, and thus violates The Principle Of Least Surprise. If a BigDecimal was supplied to Kernel.sprintf("%f",...), one of two outcomes would be expected:

1. Either that an error was raised. (In case Kernel.sprintf code to properly handle a BigDecimal was not available)
2. Or that the full BigDecimal precision would be supported.

The current situation silently produces data loss, which is surprising if ever detected.

[headius]

Interestingly enough, both JRuby and MRI produce -1.25 for BigDecimal#to_f here. MRI rounds that to -1.2 (or 1.2 for the positive form) and we round to -1.3.

MRI does the same float conversion we do on the way to formatting the value, so if you dislike that you'll have to get MRI folks involved.

[headius]

Ok actually it doesn't look like MRI does anything at all; the rounding is done by snprintf.

[headius]

This post and others like it point out that often this peculiar rounding behavior is due to inaccurate representation of certain decimals in IEEE754: http://www.perlmonks.org/?node_id=991469

However this case is not a precision issue. If the value were 1.35, however, the rounding changes for that reason:

[] ~/projects/ruby $ ruby23 -rbigdecimal -e 'p "%3.30f" % 1.25'
"1.250000000000000000000000000000"

[] ~/projects/ruby $ ruby23 -rbigdecimal -e 'p "%3.30f" % 1.35'
"1.350000000000000088817841970013"

[] ~/projects/ruby $ ruby23 -rbigdecimal -e 'p "%3.1f" % 1.25'
"1.2"

[] ~/projects/ruby $ ruby23 -rbigdecimal -e 'p "%3.1f" % 1.35'
"1.4"

This all boils down to sprintf behavior, but I'm having trouble finding definitive information on how/why sprintf rounds half down rather than up.

[headius]

Oh btw...the simple answer to "why doesn't MRI format the BigDecimal directly" is probably "*printf doesn't know anything about BigDecimal."

I'll fix the other issue for JRuby by rounding half to even in the double-related sprintf formats.

[headius]

It turns out that we tried to emulate the rounding behavior using this logic in Sprintf.java:

    private static int round(byte[] bytes, int nDigits, int roundPos, boolean roundDown) {
        int next = roundPos + 1;
        if (next >= nDigits || bytes[next] < '5' ||
                // MRI rounds up on nnn5nnn, but not nnn5 --
                // except for when they do
                (roundDown && bytes[next] == '5' && next == nDigits - 1)) {
            return nDigits;
        }
...

This was obviously an attempt to guess at sprintf behavior by observation, but it rounds down rather than to even. I've got a patch that tries to pre-round the float before this logic, but we might be able to fix this logic too.

[headius]

Well this is frustrating. My patch seems to work, but then a test in the jruby suite failed:

  # JRUBY-4802
  def test_sprintf_float
    assert_equal "0.00000", Kernel.sprintf("%.5f", 0.000004)
    assert_equal "0.00001", Kernel.sprintf("%.5f", 0.000005)
    assert_equal "0.00001", Kernel.sprintf("%.5f", 0.000006)
  end

The middle case fails because we round it to even, so it formats as 0.00000. MRI rounds it up because 0.000005 is represented inexactly as slightly more than 0.000005:

$ ruby23 -e 'p Kernel.sprintf("%.50f", 0.000005)'
"0.00000500000000000000040901526957015654772931156913"

JRuby rounds to even because it seems like our 0.000005 does get represented exactly, or at least formatting it reveals no inexactness:

$ jruby -e 'p Kernel.sprintf("%.50f", 0.000005)'
5.0E-6
0.000005
"0.00000500000000000000000000000000000000000000000000"

The first two values here are output I added to our sprintf logic: the first is a simple println of 0.000005 and the second is a NumberFormat-produced representation of the value. In all three cases no inexactness can be found.

[headius]

I had to revert the change from #4159 because it introduced new failures. This still needs to be fixed.

[aschmied]

It looks like this is fixed by bc2ba39

[kares]

on top of ^^ ... this should be handled right in 9.2.6 (due #5580) :

bin/jruby -rbigdecimal  -e 'p "%3.1f" % BigDecimal.new("-1.2499999999999999")'
"-1.2"