Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One digit short to correctly stringify a double #15119

Open
p5pRT opened this issue Jan 6, 2016 · 31 comments
Open

One digit short to correctly stringify a double #15119

p5pRT opened this issue Jan 6, 2016 · 31 comments

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Jan 6, 2016

Migrated from rt.perl.org#127182 (status was 'open')

Searchable as RT127182$

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 6, 2016

From dankogai@dan.co.jp

Created by dankogai@cpan.org

According to​:

  http​://stackoverflow.com/questions/4738768/printing-double-without-losing-precision

we need 16 decimal digits to restore a double. But Perl as of 5.22.1 gives us only 15.

  % perl -E 'say log(2)'
  0.693147180559945

As a result, we get​:

  % perl -E 'say 0+(log(2)==0.693147180559945)'
  0
  % perl -E 'say 0+(log(2)==0.6931471805599453)'
  1
  % perl -E 'say sprintf"%a",0.693147180559945'
  0x1.62e42fefa39ecp-1
  % perl -E 'say sprintf"%a",0.6931471805599453'
  0x1.62e42fefa39efp-1

FYI here's what other platforms give​:

Go -- OK
  http​://ideone.com/tzmMyb
Haskell -- OK
  http​://ideone.com/XLYEhN
JavaScript(node) -- OK
  % node -e 'console.log(Math.log(2))'
  0.6931471805599453
Perl6(rakudo latest) -- NOT OK
  % perl6 -e 'log(2).say'
  0.693147180559945
PHP5 -- NOT OK
  % php -r 'echo log(2);'
  0.69314718055995
Python2 -- NOT OK
  % python2.7 -c 'from math import *; print(log(2))'
  0.69314718056
Python3 -- OK
  % python3.5 -c 'from math import *; print(log(2))'
  0.6931471805599453
Ruby -- OK
  % ruby -e 'puts Math.log(2)'
  0.6931471805599453
Swift -- NOT OK
  http​://swiftlang.ng.bluemix.net/#/repl/0d2a70354920adef1cbee0dbe6750d012925c495ba323fdd55372614b3acc9d5

Though we should not use decimal notations to exchange doubles (and we have hexfloat support since 5.22), we can hardly avoid that with JSON is so ubiquitous and all numbers are doubles therein.

Dan the Bit-Picking Perl Monger

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.22.1:

Configured by dankogai at Thu Dec 17 00:32:07 JST 2015.

Summary of my perl5 (revision 5 version 22 subversion 1) configuration:

  Platform:
    osname=darwin, osvers=15.2.0, archname=darwin-thread-multi-2level
    uname='darwin dan-imac151.local 15.2.0 darwin kernel version 15.2.0: fri nov 13 19:56:56 pst 2015; root:xnu-3248.20.55~2release_x86_64 x86_64 '
    config_args='-DDEBUGGING -Doptimize=-g -pipe -Os -Accflags=-DPERL_USE_SAFE_PUTENV -Duseshplib -Dusethreads -Dusemultiplicity -Duse64bitall -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-common -DPERL_DARWIN -arch x86_64 -DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/opt/local/include',
    optimize='-g -pipe -Os',
    cppflags='-arch x86_64 -fno-common -DPERL_DARWIN -arch x86_64 -DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/opt/local/include'
    ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc -arch x86_64', ldflags =' -arch x86_64 -fstack-protector-strong -L/usr/local/lib -L/opt/local/lib'
    libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /usr/lib /usr/local/lib /opt/local/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=, so=dylib, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib -fstack-protector-strong'



@INC for perl 5.22.1:
    /usr/local/lib/perl5/site_perl/5.22.1/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.22.1
    /usr/local/lib/perl5/5.22.1/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.22.1
    /usr/local/lib/perl5/site_perl/5.22.0/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.22.0
    /usr/local/lib/perl5/site_perl
    .


Environment for perl 5.22.1:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dankogai
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dankogai/sbin:/Users/dankogai/bin:/Users/dankogai/.rakudobrew/bin:/Users/dankogai/node_modules/.bin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    PERL_BADLANG (unset)
    SHELL=/bin/tcsh


@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 6, 2016

From dankogai@dan.co.jp

Seems like the simplest solution is just sprintf "%.17g". Perl (and Swift) appears to be using the equivalent of sprintf "%.16g", which is one digit short.
Here're samples I wrote​:

  C http​://ideone.com/5Dq4ip
  C++ http​://ideone.com/FOT1cS
  Ruby http​://ideone.com/RVRwGy
  Perl 5.22
  #!/usr/bin/env perl
  use v5.22;
  use warnings;
  my $finish = shift || 2.0;
  die if +$finish == 0;
  my $format = '%.17g';
  my $d = $finish / 2;
  my $f = $d;
  while (1) {
  my $s = sprintf $format, $d;
  say sprintf "%s = %a", $s, $d;
  die sprintf "+($s) != %a", $d if $s != $d;
  last if $d == $finish;
  $f /= 2, $d += $f
  }
  __END__

Dan the Bug Reporter

On Wed Jan 06 03​:24​:54 2016, dankogai wrote​:

This is a bug report for perl from dankogai@​cpan.org,
generated with the help of perlbug 1.40 running under perl 5.22.1.

-----------------------------------------------------------------
[Please describe your issue here]

According to​:

http​://stackoverflow.com/questions/4738768/printing-double-without-
losing-precision

we need 16 decimal digits to restore a double. But Perl as of 5.22.1
gives us only 15.

% perl -E 'say log(2)'
0.693147180559945

As a result, we get​:

% perl -E 'say 0+(log(2)==0.693147180559945)'
0
% perl -E 'say 0+(log(2)==0.6931471805599453)'
1
% perl -E 'say sprintf"%a",0.693147180559945'
0x1.62e42fefa39ecp-1
% perl -E 'say sprintf"%a",0.6931471805599453'
0x1.62e42fefa39efp-1

FYI here's what other platforms give​:

Go -- OK
http​://ideone.com/tzmMyb
Haskell -- OK
http​://ideone.com/XLYEhN
JavaScript(node) -- OK
% node -e 'console.log(Math.log(2))'
0.6931471805599453
Perl6(rakudo latest) -- NOT OK
% perl6 -e 'log(2).say'
0.693147180559945
PHP5 -- NOT OK
% php -r 'echo log(2);'
0.69314718055995
Python2 -- NOT OK
% python2.7 -c 'from math import *; print(log(2))'
0.69314718056
Python3 -- OK
% python3.5 -c 'from math import *; print(log(2))'
0.6931471805599453
Ruby -- OK
% ruby -e 'puts Math.log(2)'
0.6931471805599453
Swift -- NOT OK
http​://swiftlang.ng.bluemix.net/#/repl/0d2a70354920adef1cbee0dbe6750d012925c495ba323fdd55372614b3acc9d5

Though we should not use decimal notations to exchange doubles (and we
have hexfloat support since 5.22), we can hardly avoid that with JSON
is so ubiquitous and all numbers are doubles therein.

Dan the Bit-Picking Perl Monger

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags​:
category=core
severity=medium
---
Site configuration information for perl 5.22.1​:

Configured by dankogai at Thu Dec 17 00​:32​:07 JST 2015.

Summary of my perl5 (revision 5 version 22 subversion 1)
configuration​:

Platform​:
osname=darwin, osvers=15.2.0, archname=darwin-thread-multi-2level
uname='darwin dan-imac151.local 15.2.0 darwin kernel version 15.2.0​:
fri nov 13 19​:56​:56 pst 2015; root​:xnu-3248.20.55~2release_x86_64
x86_64 '
config_args='-DDEBUGGING -Doptimize=-g -pipe -Os -Accflags=-
DPERL_USE_SAFE_PUTENV -Duseshplib -Dusethreads -Dusemultiplicity
-Duse64bitall -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='cc', ccflags ='-fno-common -DPERL_DARWIN -arch x86_64
-DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-
protector-strong -I/opt/local/include',
optimize='-g -pipe -Os',
cppflags='-arch x86_64 -fno-common -DPERL_DARWIN -arch x86_64
-DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-
protector-strong -I/opt/local/include'
ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.2 (clang-
700.1.81)', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16,
longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc -arch x86_64', ldflags ='
-arch x86_64 -fstack-protector-strong -L/usr/local/lib
-L/opt/local/lib'
libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib /usr/local/lib /opt/local/lib
libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
perllibs=-lpthread -ldl -lm -lutil -lc
libc=, so=dylib, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-L/usr/local/lib -L/opt/local/lib -fstack-protector-strong'

---
@​INC for perl 5.22.1​:
/usr/local/lib/perl5/site_perl/5.22.1/darwin-thread-multi-2level
/usr/local/lib/perl5/site_perl/5.22.1
/usr/local/lib/perl5/5.22.1/darwin-thread-multi-2level
/usr/local/lib/perl5/5.22.1
/usr/local/lib/perl5/site_perl/5.22.0/darwin-thread-multi-2level
/usr/local/lib/perl5/site_perl/5.22.0
/usr/local/lib/perl5/site_perl
.

---
Environment for perl 5.22.1​:
DYLD_LIBRARY_PATH (unset)
HOME=/Users/dankogai
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/Users/dankogai/sbin​:/Users/dankogai/bin​:/Users/dankogai/.rakudobrew/bin​:/Users/dankogai/node_modules/.bin​:/usr/local/bin​:/opt/local/sbin​:/opt/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/bin
PERL_BADLANG (unset)
SHELL=/bin/tcsh

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 6, 2016

From [Unknown Contact. See original ticket]

Seems like the simplest solution is just sprintf "%.17g". Perl (and Swift) appears to be using the equivalent of sprintf "%.16g", which is one digit short.
Here're samples I wrote​:

  C http​://ideone.com/5Dq4ip
  C++ http​://ideone.com/FOT1cS
  Ruby http​://ideone.com/RVRwGy
  Perl 5.22
  #!/usr/bin/env perl
  use v5.22;
  use warnings;
  my $finish = shift || 2.0;
  die if +$finish == 0;
  my $format = '%.17g';
  my $d = $finish / 2;
  my $f = $d;
  while (1) {
  my $s = sprintf $format, $d;
  say sprintf "%s = %a", $s, $d;
  die sprintf "+($s) != %a", $d if $s != $d;
  last if $d == $finish;
  $f /= 2, $d += $f
  }
  __END__

Dan the Bug Reporter

On Wed Jan 06 03​:24​:54 2016, dankogai wrote​:

This is a bug report for perl from dankogai@​cpan.org,
generated with the help of perlbug 1.40 running under perl 5.22.1.

-----------------------------------------------------------------
[Please describe your issue here]

According to​:

http​://stackoverflow.com/questions/4738768/printing-double-without-
losing-precision

we need 16 decimal digits to restore a double. But Perl as of 5.22.1
gives us only 15.

% perl -E 'say log(2)'
0.693147180559945

As a result, we get​:

% perl -E 'say 0+(log(2)==0.693147180559945)'
0
% perl -E 'say 0+(log(2)==0.6931471805599453)'
1
% perl -E 'say sprintf"%a",0.693147180559945'
0x1.62e42fefa39ecp-1
% perl -E 'say sprintf"%a",0.6931471805599453'
0x1.62e42fefa39efp-1

FYI here's what other platforms give​:

Go -- OK
http​://ideone.com/tzmMyb
Haskell -- OK
http​://ideone.com/XLYEhN
JavaScript(node) -- OK
% node -e 'console.log(Math.log(2))'
0.6931471805599453
Perl6(rakudo latest) -- NOT OK
% perl6 -e 'log(2).say'
0.693147180559945
PHP5 -- NOT OK
% php -r 'echo log(2);'
0.69314718055995
Python2 -- NOT OK
% python2.7 -c 'from math import *; print(log(2))'
0.69314718056
Python3 -- OK
% python3.5 -c 'from math import *; print(log(2))'
0.6931471805599453
Ruby -- OK
% ruby -e 'puts Math.log(2)'
0.6931471805599453
Swift -- NOT OK
http​://swiftlang.ng.bluemix.net/#/repl/0d2a70354920adef1cbee0dbe6750d012925c495ba323fdd55372614b3acc9d5

Though we should not use decimal notations to exchange doubles (and we
have hexfloat support since 5.22), we can hardly avoid that with JSON
is so ubiquitous and all numbers are doubles therein.

Dan the Bit-Picking Perl Monger

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags​:
category=core
severity=medium
---
Site configuration information for perl 5.22.1​:

Configured by dankogai at Thu Dec 17 00​:32​:07 JST 2015.

Summary of my perl5 (revision 5 version 22 subversion 1)
configuration​:

Platform​:
osname=darwin, osvers=15.2.0, archname=darwin-thread-multi-2level
uname='darwin dan-imac151.local 15.2.0 darwin kernel version 15.2.0​:
fri nov 13 19​:56​:56 pst 2015; root​:xnu-3248.20.55~2release_x86_64
x86_64 '
config_args='-DDEBUGGING -Doptimize=-g -pipe -Os -Accflags=-
DPERL_USE_SAFE_PUTENV -Duseshplib -Dusethreads -Dusemultiplicity
-Duse64bitall -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='cc', ccflags ='-fno-common -DPERL_DARWIN -arch x86_64
-DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-
protector-strong -I/opt/local/include',
optimize='-g -pipe -Os',
cppflags='-arch x86_64 -fno-common -DPERL_DARWIN -arch x86_64
-DPERL_USE_SAFE_PUTENV -DDEBUGGING -fno-strict-aliasing -pipe -fstack-
protector-strong -I/opt/local/include'
ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.2 (clang-
700.1.81)', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16,
longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc -arch x86_64', ldflags ='
-arch x86_64 -fstack-protector-strong -L/usr/local/lib
-L/opt/local/lib'
libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib /usr/local/lib /opt/local/lib
libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
perllibs=-lpthread -ldl -lm -lutil -lc
libc=, so=dylib, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-L/usr/local/lib -L/opt/local/lib -fstack-protector-strong'

---
@​INC for perl 5.22.1​:
/usr/local/lib/perl5/site_perl/5.22.1/darwin-thread-multi-2level
/usr/local/lib/perl5/site_perl/5.22.1
/usr/local/lib/perl5/5.22.1/darwin-thread-multi-2level
/usr/local/lib/perl5/5.22.1
/usr/local/lib/perl5/site_perl/5.22.0/darwin-thread-multi-2level
/usr/local/lib/perl5/site_perl/5.22.0
/usr/local/lib/perl5/site_perl
.

---
Environment for perl 5.22.1​:
DYLD_LIBRARY_PATH (unset)
HOME=/Users/dankogai
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/Users/dankogai/sbin​:/Users/dankogai/bin​:/Users/dankogai/.rakudobrew/bin​:/Users/dankogai/node_modules/.bin​:/usr/local/bin​:/opt/local/sbin​:/opt/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/bin
PERL_BADLANG (unset)
SHELL=/bin/tcsh

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 6, 2016

From @craigberry

On Wed, Jan 6, 2016 at 5​:24 AM, Dan Kogai <perlbug-followup@​perl.org> wrote​:

According to​:

http&#8203;://stackoverflow\.com/questions/4738768/printing\-double\-without\-losing\-precision

we need 16 decimal digits to restore a double. But Perl as of 5.22.1 gives us only 15.

A lossless round trip to character and back takes 17 digits, at least
according to the Wikipedia summary of the IEEE standard​:

<https://en.wikipedia.org/wiki/Double-precision_floating-point_format>

And changing 16 to 17 fixed a loss of precision problem I was having
with freetds​:

<http​://lists.ibiblio.org/pipermail/freetds/2014q3/029003.html>

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 6, 2016

The RT System itself - Status changed from 'new' to 'open'

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 7, 2016

From zefram@fysh.org

Craig A. Berry wrote​:

A lossless round trip to character and back takes 17 digits,

and also requires fixing [perl #41202].

-zefram

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 8, 2016

From @sisyphus

On Wed Jan 06 14​:17​:56 2016, craig.a.berry@​gmail.com wrote​:

A lossless round trip to character and back takes 17 digits, at least
according to the Wikipedia summary of the IEEE standard​:

<https://en.wikipedia.org/wiki/Double-precision_floating-point_format>

Similarly we find that long double and __float128 NVs are also a few bits short of satisfying the condition being considered here.

The number of decimal digits required is given by the expression 1+ceil(P*log(2)/log(10)), where P is the precision (in bits) of the NV.
P will be one of 53 , 64, 106 (double-double), 113 (__float128) - with the number of decimal digits corresponding to those values being 17, 21, 33 and 36 respectively.

Currently we get 15 (DBL_DIG), 18 (LDBL_DIG), 30 (LDBL_DIG - 1) and 32 (FLT128_DIG - 1) decimal digits (respectively).

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 8, 2016

From @sisyphus

On Sun Feb 07 18​:57​:42 2016, sisyphus wrote​:

Similarly we find that long double and __float128 NVs are also a few
bits short of satisfying the condition being considered here.

Groan ... I did say "a few bits short", but I meant, of course, "a few decimal digits short".

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 13, 2016

From @jhi

I quickly tried adding the number of decimal digits in the default stringification, as suggested by Sisyphus, and one thing that immediately fails is the "classic" illusion that 0.1 stringifies as "0.1".

And t/base/num.t tests for that, and other similar illusions. Whether it is right to be testing such, is of course another matter. I think it's quite dubious, anyway, given how tricksy the binary<->decimal conversion are. FWIW, I seem to have been part of the dubiousness​:

http​://perl5.git.perl.org/perl.git/commit/925fa5a876ae65e605ff8becef8ac0c232c8148f?f=t/base/num.t
http​://perl5.git.perl.org/perl.git/commit/0ad373b6ac664c739ef27968c44a6d6a5387a7ab?f=t/base/num.t

I temporarily bypassed those failing t/base tests, and the following still fail in "make minitest"​:

Failed 4 tests out of 327, 98.78% okay.
  op/hexfp.t
  op/override.t
  op/pack.t
  op/tie.t

(details below)

I'll work through those (probably just similarly bypassing them for now) and see how the full "make test" goes. One thing I know will be all fireworks is Math​::Complex which assumes a certain number of fractional digits.

Failure details​:

t/base/num.t​:

not ok 5 # 0.10000000000000001
not ok 6 # -0.10000000000000001
not ok 7 # 0.10000000000000001
not ok 8 # -0.10000000000000001
not ok 34 # 9.0000000000000006e-05
not ok 35 # 1.1000000000000001
not ok 37 # 1.0009999999999999
not ok 39 # 1.0000100000000001
not ok 40 # 1.0000009999999999
not ok 45 # 1.0000000000000001e+34

t/op/hexfp.t​:

# Failed test 95 - at t/op/hexfp.t line 214
# got "4503599627370495"
# expected "4503599627370500"
# Failed test 97 - at t/op/hexfp.t line 219
# got "4503599627370495.5"
# expected "4503599627370500"
# Failed test 99 - at t/op/hexfp.t line 224
# got "4503599627370496"
# expected "4503599627370500"
# Failed test 101 - at t/op/hexfp.t line 229
# got "15.999999999999996"
# expected "16"
# Failed test 103 - at t/op/hexfp.t line 234
# got "15.999999999999998"
# expected "16"
not ok 95
not ok 97
not ok 99
not ok 101
not ok 103

t/op/override.t​:

# Failed test 6 - at t/op/override.t line 43
# got "5.0060000000000002"
# expected "5.006"
# Failed test 10 - at t/op/override.t line 58
# got ' 5.0060000000000002 '
# expected /(?^​: 5\.006 )/
not ok 6
not ok 10

t/op/pack.t​:

# Failed test 13177 - at t/op/pack.t line 1418
# got "173 1.283476517e-45 42 215 173 1.283476517e-45 42 215"
# expected "173 1.2834765169999999e-45 42 215 173 1.2834765169999999e-45 42 215"
# Failed test 13180 - at t/op/pack.t line 1431
# got "173 1.283476517e-45 42 215"
# expected "173 1.2834765169999999e-45 42 215"
not ok 13177

t/op/tie.t​:

not ok 18 # TODO IO "self-tie" via TEMP glob
# Failed test 18 - at t/op/tie.t line 18
PROG​:
sub TIESCALAR { bless {} }
sub FETCH { my $x = 3.3; 1 if 0+$x; $x }
tie $h, "main";
print $h,"\n";
EXPECTED​:
3.3
GOT​:
3.2999999999999998
not ok 33
# Failed test 33 - at t/op/tie.t line 18

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @jhi

Some further results.

TL;DR​: This may be a too deep a pool to wade in this close to 5.24.

I think the best technical summary is​:

./perl -wle 'print .1'
0.10000000000000001

Find attached​:
- two patches that change the number of output digits
- one patch that changes t/base/num.t to pass
- one patch that changes four more tests to pass

With these, "make minitest" passes for me in OS X with default settings (boring 64-bit IEEE 754, x86_64).

Also find attached​:
- make-test.log.gz
- perl-harness.log.gz

One thing that kills many of the full test ones is that the v-string stringification goes bang.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @jhi

0001-Introduce-NV_DEC_DIG.patch
From bda66201bbfde3169e888ff26cafa7bb8ba028b2 Mon Sep 17 00:00:00 2001
From: Jarkko Hietaniemi <jhi@iki.fi>
Date: Sat, 13 Feb 2016 16:22:35 -0500
Subject: [PATCH 1/4] Introduce NV_DEC_DIG.

NV_DEC_DIG is like NV_DIG (number of digits for decimal floating)
but usually 1-3 more.
---
 perl.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/perl.h b/perl.h
index a2ba5cd..050f25e 100644
--- a/perl.h
+++ b/perl.h
@@ -2065,6 +2065,25 @@ extern long double Perl_my_frexpl(long double x, int *e);
 #       endif
 #   endif
 
+/* NV_DEC_DIG is the number of decimal digits required to print
+ * NV_MANT_DIG bits of precision in decimal, without loss of
+ * precision.  This is usually 1-3 more digits than NV_DIG.
+ * We have precomputed values for the most common cases,
+ * and generic formula for the rest. */
+#ifndef NV_DEC_DIG
+#  if NV_MANT_DIG == 53 /* IEEE 754 64-bit double */
+#    define NV_DEC_DIG 17
+#  elif NV_MANT_DIG == 64 /* x86 80-bit extended precision */
+#    define NV_DEC_DIG 21
+#  elif (NV_MANT_DIG == 106 || NV_MANT_DIG == 107) /* double-double */
+#    define NV_DEC_DIG 33
+#  elif NV_MANT_DIG == 113 /* IEEE 754 128-bit double */
+#    define NV_DEC_DIG 36
+#  else
+#    defined NV_DEC_DIG (1+Perl_ceil(NV_MANT_DIG*log(2)/log(10)))
+#  endif
+#endif
+
 /* These math interfaces are C89. */
 #   define Perl_acos acos
 #   define Perl_asin asin
-- 
2.7.0

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @jhi

0002-Use-the-new-NV_DEC_DIG.patch
From 76b74fccf5d7779bd79305c9889bcb35f6ec59c0 Mon Sep 17 00:00:00 2001
From: Jarkko Hietaniemi <jhi@iki.fi>
Date: Sat, 13 Feb 2016 16:50:59 -0500
Subject: [PATCH 2/4] Use the new NV_DEC_DIG

---
 dump.c |  2 +-
 sv.c   | 12 ++++++------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/dump.c b/dump.c
index bc2776a..cbe64b8 100644
--- a/dump.c
+++ b/dump.c
@@ -1567,7 +1567,7 @@ Perl_do_sv_dump(pTHX_ I32 level, PerlIO *file, SV *sv, I32 nest, I32 maxnest, bo
 		&& type != SVt_PVIO && !isGV_with_GP(sv) && !SvVALID(sv))
 	       || type == SVt_NV) {
 	STORE_LC_NUMERIC_UNDERLYING_SET_STANDARD();
-	Perl_dump_indent(aTHX_ level, file, "  NV = %.*" NVgf "\n", NV_DIG, SvNVX(sv));
+	Perl_dump_indent(aTHX_ level, file, "  NV = %.*" NVgf "\n", NV_DEC_DIG, SvNVX(sv));
 	RESTORE_LC_NUMERIC_UNDERLYING();
     }
 
diff --git a/sv.c b/sv.c
index 819a250..fc04137 100644
--- a/sv.c
+++ b/sv.c
@@ -3103,7 +3103,7 @@ Perl_sv_2pv_flags(pTHX_ SV *const sv, STRLEN *const lp, const I32 flags)
                 size =
                     1 + /* sign */
                     1 + /* "." */
-                    NV_DIG +
+                    NV_DEC_DIG +
                     1 + /* "e" */
                     1 + /* sign */
                     5 + /* exponent digits */
@@ -3112,7 +3112,7 @@ Perl_sv_2pv_flags(pTHX_ SV *const sv, STRLEN *const lp, const I32 flags)
 
                 s = SvGROW_mutable(sv, size);
 #ifndef USE_LOCALE_NUMERIC
-                SNPRINTF_G(SvNVX(sv), s, SvLEN(sv), NV_DIG);
+                SNPRINTF_G(SvNVX(sv), s, SvLEN(sv), NV_DEC_DIG);
 
                 SvPOK_on(sv);
 #else
@@ -3127,7 +3127,7 @@ Perl_sv_2pv_flags(pTHX_ SV *const sv, STRLEN *const lp, const I32 flags)
                         s = SvGROW_mutable(sv, size);
                     }
 
-                    SNPRINTF_G(SvNVX(sv), s, SvLEN(sv), NV_DIG);
+                    SNPRINTF_G(SvNVX(sv), s, SvLEN(sv), NV_DEC_DIG);
 
                     /* If the radix character is UTF-8, and actually is in the
                      * output, turn on the UTF-8 flag for the scalar */
@@ -11295,9 +11295,9 @@ Perl_sv_vcatpvfn_flags(pTHX_ SV *const sv, const char *const pat, const STRLEN p
     const bool pat_utf8 = has_utf8; /* the pattern is in utf8? */
     SV *nsv = NULL;
     /* Times 4: a decimal digit takes more than 3 binary digits.
-     * NV_DIG: mantissa takes than many decimal digits.
+     * NV_DEC_DIG: mantissa takes than many decimal digits.
      * Plus 32: Playing safe. */
-    char ebuf[IV_DIG * 4 + NV_DIG + 32];
+    char ebuf[IV_DIG * 4 + NV_DEC_DIG + 32];
     bool no_redundant_warning = FALSE; /* did we use any explicit format parameter index? */
     bool hexfp = FALSE; /* hexadecimal floating point? */
 
@@ -11368,7 +11368,7 @@ Perl_sv_vcatpvfn_flags(pTHX_ SV *const sv, const char *const pat, const STRLEN p
                     /* Add check for digits != 0 because it seems that some
                        gconverts are buggy in this case, and we don't yet have
                        a Configure test for this.  */
-                    if (digits && digits < sizeof(ebuf) - NV_DIG - 10) {
+                    if (digits && digits < sizeof(ebuf) - NV_DEC_DIG - 10) {
                         /* 0, point, slack */
                         STORE_LC_NUMERIC_SET_TO_NEEDED();
                         SNPRINTF_G(nv, ebuf, size, digits);
-- 
2.7.0

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @jhi

0003-No-.1-and-the-like-in-t-base-num.t.patch
From 2f4f2a50230cd9278a201d9a0c09a3ed80c5f56e Mon Sep 17 00:00:00 2001
From: Jarkko Hietaniemi <jhi@iki.fi>
Date: Sun, 14 Feb 2016 10:59:16 -0500
Subject: [PATCH 3/4] No .1 and the like in t/base/num.t

Since 0.1 is a lie with IEEE 754.
---
 t/base/num.t | 44 ++++++++++++++++++++++----------------------
 1 file changed, 22 insertions(+), 22 deletions(-)

diff --git a/t/base/num.t b/t/base/num.t
index 8a61fb9..b138fbf 100644
--- a/t/base/num.t
+++ b/t/base/num.t
@@ -17,17 +17,17 @@ print $a eq "1"       ? "ok 3\n"  : "not ok 3 # $a\n";
 $a = -1.; "$a";
 print $a eq "-1"      ? "ok 4\n"  : "not ok 4 # $a\n";
 
-$a = 0.1; "$a";
-print $a eq "0.1"     ? "ok 5\n"  : "not ok 5 # $a\n";
+$a = 0.125; "$a"; # 1+1/8
+print $a eq "0.125"     ? "ok 5\n"  : "not ok 5 # $a\n";
 
-$a = -0.1; "$a";
-print $a eq "-0.1"    ? "ok 6\n"  : "not ok 6 # $a\n";
+$a = -0.125; "$a";
+print $a eq "-0.125"    ? "ok 6\n"  : "not ok 6 # $a\n";
 
-$a = .1; "$a";
-print $a eq "0.1"     ? "ok 7\n"  : "not ok 7 # $a\n";
+$a = .125; "$a";
+print $a eq "0.125"     ? "ok 7\n"  : "not ok 7 # $a\n";
 
-$a = -.1; "$a";
-print $a eq "-0.1"    ? "ok 8\n"  : "not ok 8 # $a\n";
+$a = -.125; "$a";
+print $a eq "-0.125"    ? "ok 8\n"  : "not ok 8 # $a\n";
 
 $a = 10.01; "$a";
 print $a eq "10.01"   ? "ok 9\n"  : "not ok 9 # $a\n";
@@ -131,26 +131,26 @@ if ($^O eq 'os2') { # In the long run, fix this.  For 5.8.0, deal.
     print $a eq "0.0001"  ? "ok 33\n" : "not ok 33 # $a\n";
 }
 
-$a = 0.00009; "$a";
-print $a eq "9e-05" || $a eq "9e-005" ? "ok 34\n"  : "not ok 34 # $a\n";
+$a = 1/16384; "$a";
+print $a eq "6.103515625e-05" || $a eq "6.103515625e-005" ? "ok 34\n"  : "not ok 34 # $a\n";
 
-$a = 1.1; "$a";
-print $a eq "1.1"     ? "ok 35\n" : "not ok 35 # $a\n";
+$a = 1.125; "$a";
+print $a eq "1.125"   ? "ok 35\n" : "not ok 35 # $a\n";
 
 $a = 1.01; "$a";
 print $a eq "1.01"    ? "ok 36\n" : "not ok 36 # $a\n";
 
-$a = 1.001; "$a";
-print $a eq "1.001"   ? "ok 37\n" : "not ok 37 # $a\n";
+$a = 1.0009765625; "$a";  # 1+1/1024
+print $a eq "1.0009765625" ? "ok 37\n" : "not ok 37 # $a\n";
 
-$a = 1.0001; "$a";
-print $a eq "1.0001"  ? "ok 38\n" : "not ok 38 # $a\n";
+$a = 1.0001220703125; "$a"; # 1+1/8192
+print $a eq "1.0001220703125" ? "ok 38\n" : "not ok 38 # $a\n";
 
-$a = 1.00001; "$a";
-print $a eq "1.00001" ? "ok 39\n" : "not ok 39 # $a\n";
+$a = 1.0000152587890625; "$a"; # 1+1/65536
+print $a eq "1.0000152587890625" ? "ok 39\n" : "not ok 39 # $a\n";
 
-$a = 1.000001; "$a";
-print $a eq "1.000001" ? "ok 40\n" : "not ok 40 # $a\n";
+$a = 1.0000009536743164; "$a"; # 1+1/1048576
+print $a eq "1.0000009536743164" ? "ok 40\n" : "not ok 40 # $a\n";
 
 $a = 0.; "$a";
 print $a eq "0"       ? "ok 41\n" : "not ok 41 # $a\n";
@@ -164,9 +164,9 @@ print $a eq "-100000" ? "ok 43\n" : "not ok 43 # $a\n";
 $a = 123.456; "$a";
 print $a eq "123.456" ? "ok 44\n" : "not ok 44 # $a\n";
 
-$a = 1e34; "$a";
+$a = 1e30; "$a";
 unless ($^O eq 'posix-bc')
-{ print $a eq "1e+34" || $a eq "1e+034" ? "ok 45\n" : "not ok 45 # $a\n"; }
+{ print $a eq "1e+30" || $a eq "1e+030" ? "ok 45\n" : "not ok 45 # $a\n"; }
 else
 { print "ok 45 # skipped on $^O\n"; }
 
-- 
2.7.0

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @jhi

0004-Binarily-exact-floats-handle-.5999-vs-.6000-diffs.patch
From b0e300d922ebfb1569430e920e8d686d95157ba0 Mon Sep 17 00:00:00 2001
From: Jarkko Hietaniemi <jhi@iki.fi>
Date: Sun, 14 Feb 2016 11:36:51 -0500
Subject: [PATCH 4/4] Binarily exact floats, handle .5999 vs .6000 diffs.

---
 t/op/hexfp.t    | 10 +++++-----
 t/op/override.t |  9 +++++++--
 t/op/pack.t     | 14 ++++++++++----
 t/op/tie.t      |  4 ++--
 4 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/t/op/hexfp.t b/t/op/hexfp.t
index 4b2a96d..7d05f4f 100644
--- a/t/op/hexfp.t
+++ b/t/op/hexfp.t
@@ -211,27 +211,27 @@ SKIP:
         undef $a;
         eval '$a = 0xfffffffffffffp0';  # 52 bits.
         is(get_warn(), undef);
-        is($a, 4.5035996273705e+15);
+        is($a, 4503599627370495);
 
         undef $a;
         eval '$a = 0xfffffffffffff.8p0';  # 53 bits.
         is(get_warn(), undef);
-        is($a, 4.5035996273705e+15);
+        is($a, 4503599627370495.5);
 
         undef $a;
         eval '$a = 0xfffffffffffff.cp0';  # 54 bits.
         like(get_warn(), qr/^Hexadecimal float: mantissa overflow/);
-        is($a, 4.5035996273705e+15);
+        is($a, 4503599627370496);
 
         undef $a;
         eval '$a = 0xf.ffffffffffffp0';  # 52 bits.
         is(get_warn(), undef);
-        is($a, 16);
+        is($a, 15.999999999999996);
 
         undef $a;
         eval '$a = 0xf.ffffffffffff8p0';  # 53 bits.
         is(get_warn(), undef);
-        is($a, 16);
+        is($a, 15.999999999999998);
 
         undef $a;
         eval '$a = 0xf.ffffffffffffcp0';  # 54 bits.
diff --git a/t/op/override.t b/t/op/override.t
index e660311..ffee721 100644
--- a/t/op/override.t
+++ b/t/op/override.t
@@ -28,7 +28,10 @@ is( 45, time + 3 );
 # require has special behaviour
 #
 my $r;
-BEGIN { *CORE::GLOBAL::require = sub { $r = shift; 1; } }
+sub drop_zero_tail {
+    $_[0] =~ s/(\.00\d)0{6,}\d/\1/;
+}
+BEGIN { *CORE::GLOBAL::require = sub { $r = shift; drop_zero_tail($r); 1; } }
 
 require Foo;
 is( $r, "Foo.pm" );
@@ -55,7 +58,9 @@ is( $r, join($dirsep, "Foo", "Bar.pm") );
     my @r;
     local *CORE::GLOBAL::require = sub { push @r, shift; 1; };
     eval "use 5.006";
-    like( " @r ", qr " 5\.006 " );
+    my $r = " @r ";
+    drop_zero_tail($r);
+    like( $r, qr " 5\.006 " );
 }
 
 {
diff --git a/t/op/pack.t b/t/op/pack.t
index a2da636..70c2777 100644
--- a/t/op/pack.t
+++ b/t/op/pack.t
@@ -1413,9 +1413,12 @@ is(scalar unpack('A /A /A Z20', '3004bcde'), 'bcde');
   ok( length $p);
   my @b = unpack "$t X[$t] $t", $p;	# Extract, step back, extract again
   is(scalar @b, 2 * scalar @a);
+  my $a = "@a @a";
   $b = "@b";
-  $b =~ s/(?:17000+|16999+)\d+(e-45) /17$1 /gi; # stringification is gamble
-  is($b, "@a @a");
+  for ($a, $b) {
+      s/(?:170*|16999+)\d+(e-45) /17$1 /gi; # stringification is gamble
+  }
+  is($b, $a);
 
   use warnings qw(NONFATAL all);;
   my $warning;
@@ -1426,9 +1429,12 @@ is(scalar unpack('A /A /A Z20', '3004bcde'), 'bcde');
 
   is($warning, undef);
   is(scalar @b, scalar @a);
+  my $a = "@a";
   $b = "@b";
-  $b =~ s/(?:17000+|16999+)\d+(e-45) /17$1 /gi; # stringification is gamble
-  is($b, "@a");
+  for ($a, $b) {
+      s/(?:17000+|16999+)\d+(e-45) /17$1 /gi; # stringification is gamble
+  }
+  is($b, $a);
 }
 
 is(length(pack("j", 0)), $Config{ivsize});
diff --git a/t/op/tie.t b/t/op/tie.t
index ae0db6f..51131e7 100644
--- a/t/op/tie.t
+++ b/t/op/tie.t
@@ -563,11 +563,11 @@ FIRSTKEY
 empty
 ########
 sub TIESCALAR { bless {} }
-sub FETCH { my $x = 3.3; 1 if 0+$x; $x }
+sub FETCH { my $x = 3.25; 1 if 0+$x; $x }
 tie $h, "main";
 print $h,"\n";
 EXPECT
-3.3
+3.25
 ########
 sub TIESCALAR { bless {} }
 sub FETCH { shift()->{i} ++ }
-- 
2.7.0

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2016

From @sisyphus

-----Original Message-----
From​: Jarkko Hietaniemi via RT
Sent​: Monday, February 15, 2016 11​:30 AM
To​: OtherRecipients of perl Ticket #127182​:
Cc​: perl5-porters@​perl.org
Subject​: [perl #127182] One digit short to correctly stringify a double

I think the best technical summary is​:

./perl -wle 'print .1'
0.10000000000000001

Additionally​:

perl -wle 'print "ok" if 0.1 == 0.10000000000000001;'
ok

perl -wle 'print "ok" if 0.1 == 0.10000000000000000;'
ok

and​:

perl -wle 'print "ok" if sqrt 2.0 == 1.4142135623730951;'
ok

perl -wle 'print "ok" if sqrt 2.0 == 1.4142135623730952;'
ok

I'm in favour of this change (for 5.26) - it makes good sense to me (despite
the anomalies it allows one to concoct).
But can we take it for granted that it's what we should be doing ?

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 16, 2016

From @epa

The original bug report contains examples of languages that stringify
doubles correctly, with sixteen decimal digits. For example Python 3 and
Ruby are OK.
But I note that these languages on printing (1.0 / 10.0) output just '0.1'.
What heuristic are they using to output that instead of 0.10000000000000001?

Both 0.1 and 0.10000000000000001 represent the same double​:

% perl -E '$x = 0.1; $y = 0.10000000000000001; say $x == $y'
1

Should perl start with 16 d.p., and then if the resulting string does indeed
have sixteen digits after the decimal point, also try some smaller number of
d.p. and see if the resulting shorter string parses back to the same double?
If so, the shorter string would be output.

With the current number of d.p. output by perl, are there any surprising cases
where a double is output as a string but there is another, shorter, string
which parses to the same double?

--
Ed Avis <eda@​waniasset.com>

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 16, 2016

From zefram@fysh.org

Ed Avis wrote​:

What heuristic are they using to output that instead of 0.10000000000000001?

Probably outputting the minimum number of digits such that reading
the output back in will exactly recreate the NV. So for example the
NV that you get from the input "0.1" will be output as "0.1", but the
adjacent NVs will output about sixteen decimal places. For algorithm,
look up the paper by Guy Steele. Steele likes this heuristic, and wrote
a requirement for it into the Java spec. I like this heuristic too,
and think it would be a good idea for Perl to apply it.

In our case, this heuristic runs into a complication due to the
long-standing [perl #41202]. The intent of the heuristic, and the
details of Steele's algorithm, are founded on an assumption that the
input text->float algorithm is correct, in the sense that (other than on
overflow) it yields the NV whose value is closest to the mathematical
value represented by the decimal input. Due to our bug, that is not
the case in Perl.

I think we should fix the bug. But pending that fix, we have to decide
what version of the output heuristic we should apply. I think it would be
madness to make the output algorithm cater to the buggy input algorithm,
because that would mean generating output that is not even correctly
rounded to the precision that it shows. We should generate output such
that *a correct input algorithm* would recreate the NV, even though
that means we (still) wouldn't have practical round-tripping for the
time being.

-zefram

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 16, 2016

From @jhi

I think it would be madness to make the output algorithm cater to the buggy
input algorithm, because that would mean generating output that is not even correctly
rounded to the precision that it shows. We should generate output such
that *a correct input algorithm* would recreate the NV, even though
that means we (still) wouldn't have practical round-tripping for the
time being.

Please see

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122482

-zefram

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 16, 2016

From @epa

I agree that #122482 looks worth fixing, and in an ideal world it might be
fixed first before this bug is addressed. However, it's not a hard
dependency. When stringifying a double and using Steele's algorithm to pick
the best (shortest) decimal representation, that can be done assuming a
correct string->double parser.

This does mean that, with Perl's current parser, there would be floating point
values that can't be round-tripped to strings and back again. But that is the
case at the moment anyway, so could hardly be considered a regression.
Correct round-tripping would start to happen once both bugs are fixed.

Anyway - what I really wanted to say was that the change to output 16 d.p.
naively, causing 1.0/10.0 to become 0.10000000000000001, is probably not
a good idea to apply. It would cause too much hassle and too many questions,
even if it can be justified on technical grounds.

--
Ed Avis <eda@​waniasset.com>

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 17, 2016

From @sisyphus

-----Original Message-----
From​: Ed Avis

Anyway - what I really wanted to say was that the change to output 16 d.p.
naively, causing 1.0/10.0 to become 0.10000000000000001, is probably not a
good idea to apply.

My main reason for disagreeing is that the mpfr C library outputs
"0.10000000000000001" (actually, the equivalent "1.0000000000000001e-1") and
I choose to take my guidance from what that library does, in preference to
what Java or Python or other languages do.
(The corollary here is that if the mpfr developers start employing the
Steele heuristic, then I'll switch camps straight away :-)

I do wonder if there's an option to pragmatise here ?
Would it be feasible to write a "steele" pragma that turns on the sort of
behaviour that Ed and Zefram are promoting with a simple "use steele;" ?

One other (obvious but hitherto unmentioned) fact is that anyone who wants
17 decimal digit output precision for doubles can already get that via
printf(). That capability is already there - do we really need to do
anything ?

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 17, 2016

From @jhi

FWIW I updated https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122482 with some new pointers to munch with your breakfast cereal.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 17, 2016

From @sisyphus

-----Original Message-----
From​: Jarkko Hietaniemi via RT
Sent​: Wednesday, February 17, 2016 12​:50 AM
To​: OtherRecipients of perl Ticket #127182​:
Cc​: perl5-porters@​perl.org
Subject​: [perl #127182] One digit short to correctly stringify a double

Please see

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122482

Is there a cross-platform solution proposed there ?
I'm not all that familiar with netlib, and you seem to be a bit ambivalent
about it in that bug report - but if therein lies a solution to this
longstanding problem then I say "go for it".

I once thought that using C's strtod/strtold/strtoflt128 would be the way to
go, but I've since struck bugs in them that have changed my mind.

On Windows, I found that mingw.org's gcc-4.7.0/runtime had a strtod() that
was prone to making incorrect assignments. When I pointed this out a coupla
years ago, their lead developer (Keith Marshall) took exception to my use of
the word "bug", said that he had no intention of doing anything about it,
and suggested that if it bothered me then I should use a designated math C
library.
First thing I did was to take his advice and refer always to the mpfr C
library.
Second thing I did was to take no further interest in anything provided by
that vendor, switching instead to mingw-w64.

Sadly, mingw-w64 also appear to be rather disinterested in doing anything
about math bugs - but at least they can assign correctly (IME).

I think my only problems with strtold have been on ppc64, gcc-4.6.3.
Not sure that I've *ever* had a problem with strtoflt128 ... I'd have to
check.

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 17, 2016

From zefram@fysh.org

sisyphus1@​optusnet.com.au wrote​:

Would it be feasible to write a "steele" pragma that turns on the
sort of behaviour that Ed and Zefram are promoting with a simple
"use steele;" ?

No. Stringification is not meaningfully subject to lexical scoping,
because objects can readily cross module boundaries with the sender
intending that the receiver will stringify. Variations in stringification
would also break the caching of stringifications.

One other (obvious but hitherto unmentioned) fact is that anyone who
wants 17 decimal digit output precision for doubles can already get
that via printf().

Yes, that's fine for people with very specific needs for textual
representation of numbers. But it's still worthwhile, and very Perlish,
to make the default stringification as good as it can be for general use.
The default stringification is the most easily accessible textual
representation of the number, so its quality should be a concern.
Also, its frequent implicit invocation is a good reason to make it as
unsurprising as possible. To have different NVs stringify identically is
surprising; to have the closest approximation to 0.1 stringify other than
"0.1" is also surprising; so I think Steele's rule specifically achieves
least surprise here.

-zefram

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 18, 2016

From @sisyphus

-----Original Message-----
From​: Zefram
Sent​: Thursday, February 18, 2016 4​:08 AM
To​: perl5-porters@​perl.org
Subject​: Re​: [perl #127182] One digit short to correctly stringify a double

To have different NVs stringify identically is surprising; to have the
closest approximation to 0.1 stringify other than "0.1" is also
surprising; so I think Steele's rule specifically achieves least surprise
here.

I keep forgetting that perl has a thing about minimising surprises - and
that's probably a good enough reason for perl to abide by Steele's rule.
So I'll now accept that I can live with that :-)

The hex form of 0.1 is 0x1.999999999999ap-4, and whenever that double is
printed, I gather Steele's rule will decree that it be printed out as
'0.1' - irrespective of whether that double was derived as 1/10 or
10000000000000001/1e17.
But there's nothing to object about in that, AFAICS.

Anyway, as I've already noted, one can always resort to printf if need be.

Cheers,
Rob

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 18, 2016

From @jhi

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122482

Is there a cross-platform solution proposed there ?

Yes, gdtoa is the widely used solution for this problem.

The 'g'-dtoa handles long doubles of all kinds, while bare dtoa does not.

As opposed to other languages' runtimes, Perl is blessed/cursed with being able to be configured/used with different doubles, so we would need much of the 'gdtoa'.

Reminder​: despite the name 'dtoa', it also does 'atod'. So not just printf/gconvert, but also strtod.

I'm not all that familiar with netlib,

See http​://netlib.org/

and you seem to be a bit ambivalent

My ambivalency stems from a few things​:
- gdtoa is not just one file, it's a full library, so it would be an integration/maintenance challenge
- the code requires quite a bit of integration help
- code bloat

At the moment I am actually thinking that a technically better plan of action would be to make the gdtoa into a standalone library (with all the enhancements/fixes made by various projects using the library), and then just make it possible for Perl to use the library (just like with quadmath). The major hurdle of this plan might be getting a good working relationship with David Gay, but he seems a bit hard/slow to connect with. (And the code is copyright Lucent, that might be another painful issue, even though the code as such has been widely used in open source projects.) And not to mention that there would be a considerable lead time before the library would be available anywhere.

So shorter term dealing with the integration pain might be unavoidable.

about it in that bug report - but if therein lies a solution to this
longstanding problem then I say "go for it".

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 20, 2016

From zefram@fysh.org

I wrote​:

                                                   For algorithm\,

look up the paper by Guy Steele.

I got bored and looked it up myself​:

Steele, Jr., Guy L., and White, Jon L., "How to print floating-point
numbers accurately", in Proc. ACM SIGPLAN Notices, vol. 25, 6 (June 1990).

https://lists.nongnu.org/archive/html/gcl-devel/2012-10/pdfkieTlklRzN.pdf

There's also some interesting historical commentary in​:

Steele, Jr., Guy L., and White, Jon L., "How to print floating-point
numbers accurately (retrospective)", in "20 Years of the ACM/SIGPLAN
Conference on Programming Language Design and Implementation (1979-1999)​:
A Selection", ACM Press, 2003.

http​://grouper.ieee.org/groups/754/email/pdfq3pavhBfih.pdf

-zefram

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Mar 14, 2016

From @karenetheridge

I just ran into something similar with PAUSE, when attempting to index a version containing a double​: andk/pause#203

Although I do not know if it is the same issue as I haven't yet determined what versions of perl and version.pm are running there.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Mar 26, 2018

From cpan@zoffix.com

On Sat, 20 Feb 2016 08​:21​:38 -0800, zefram@​fysh.org wrote​:

I wrote​:

                                                   For algorithm\,

look up the paper by Guy Steele.

I got bored and looked it up myself​:

Steele, Jr., Guy L., and White, Jon L., "How to print floating-point
numbers accurately", in Proc. ACM SIGPLAN Notices, vol. 25, 6 (June 1990).

https://lists.nongnu.org/archive/html/gcl-devel/2012-10/pdfkieTlklRzN.pdf

There's also some interesting historical commentary in​:

Steele, Jr., Guy L., and White, Jon L., "How to print floating-point
numbers accurately (retrospective)", in "20 Years of the ACM/SIGPLAN
Conference on Programming Language Design and Implementation (1979-1999)​:
A Selection", ACM Press, 2003.

http​://grouper.ieee.org/groups/754/email/pdfq3pavhBfih.pdf

-zefram

FWIW, recently worked on the same issue in Perl 6 on MoarVM, figured I'd mention
some of my findings​:

Grisu3 algo[^1] looks to be an update to the Dragon stuff in those papers.
Based on my research, Dragon4 used as fallback for the 0.5% of cases that
Grisu3 can't handle seems to be the current state of the art in the area
and is used by Rust and V8.

I made[^2] MoarVM to use Grisu3 with snprintf("%.17g") as fallback for the 0.5% of cases.
I also made a couple of modifications[^3][^4] for post-Grisu renderer that sticks the
dot in the right place.

So far the code works great, fixed a bunch of precision bugs, and made Perl 6's
Num stringification 2x faster (we used plain snprintf("%.15g") before). There's
a drift issue[^5] with some numbers, but I can't reproduce it in the standalone
Grisu3 C code, so I suspect the cause of the drift is how Perl 6 actually
parses the Num numerals and not in Grisu.

[1] https://goo.gl/cbvogg
[2] MoarVM/MoarVM@067c059
[3] MoarVM/MoarVM@af2eb8a
[4] MoarVM/MoarVM@8841c42
[5] rakudo/rakudo#1651

Cheers,
ZZ

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Mar 8, 2019

From @sisyphus

On Sun, 25 Mar 2018 20​:38​:05 -0700, cpan@​zoffix.com wrote​:

FWIW, recently worked on the same issue in Perl 6 on MoarVM, figured
I'd mention
some of my findings​:

Grisu3 algo[^1] looks to be an update to the Dragon stuff in those
papers.
Based on my research, Dragon4 used as fallback for the 0.5% of cases
that
Grisu3 can't handle seems to be the current state of the art in the
area
and is used by Rust and V8.

I made[^2] MoarVM to use Grisu3 with snprintf("%.17g") as fallback for
the 0.5% of cases.
I also made a couple of modifications[^3][^4] for post-Grisu renderer
that sticks the
dot in the right place.

So far the code works great, fixed a bunch of precision bugs, and made
Perl 6's
Num stringification 2x faster (we used plain snprintf("%.15g")
before). There's
a drift issue[^5] with some numbers, but I can't reproduce it in the
standalone
Grisu3 C code, so I suspect the cause of the drift is how Perl 6
actually
parses the Num numerals and not in Grisu.

[1] https://goo.gl/cbvogg
[2]
MoarVM/MoarVM@067c059
[3]
MoarVM/MoarVM@af2eb8a
[4]
MoarVM/MoarVM@8841c42
[5] rakudo/rakudo#1651

Cheers,
ZZ

Nice work - perl5 should have been doing the same since 10 years ago (at least).
The problem regarding perl5's propensity to assign inaccurately has pretty much been fixed as of version 5.29.4 or thereabouts - see https://rt.perl.org/Public/Bug/Display.html?id=41202 .
The fix relies on the availability of correct strtod/strtold functionality at the C level - which recent versions of gcc/libc seem to provide. (The -Dusequadmath builds have always relied on C's strtoflt128.)
But perl5's appalling disregard for accuracy of output still prevails.

The https://goo.gl/cbvogg link is not turning up anything. Do you have an updated link for that ?
I have found https://github.com/juj/MathGeoLib/blob/master/src/Math/grisu3.c. Is that what I need ?
Also, I'm a little intrigued by the "0.5% of cases that Grisu3 can't handle". What is it that characterizes that 0.5% ?

I've been playing around with the approach taken by Steele and White.
In Math​::MPFR-4.09_001 I've an implementation (nvtoa XSub) based on their "FPP2" algorithm as per page 120 of "How to Print Floating-Point Numbers Accurately". It's now much improved upon the implementation that appeared in Math-MPFR-4.09 but, AFAICT, it's not up to the performance level of perl6 - being 4 to 5 times slower according to my rough benchmarking.

My nvtoa() function accommodates -Duselongdouble (80-bit, 128-bit and DoubleDouble) and -Dusequadmath builds, as well as the more usual 'double' builds.

The mpfr library is currently only used for the DoubleDouble build - and complete removal of that mpfr library dependency is a TODO.
It has a multi-prec integer dependency, for which I'm relying on GMP.

On my 'double' and 'long double' builds "nvtoa($nv)" is currently taking about 3 times longer than perl5's sprintf("%.16e", $nv).
On my '__float128' and 'doubledouble' builds "nvtoa($nv)" is currently taking about 4 times longer than perl's sprintf(%.35e", $nv).

That's good enough for my own uses, but it's not "Rolls Royce" category ;-)

Cheers,
Rob

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.