g++ vs subnormals in 5.26.0

[p5pRT]

Migrated from rt.perl.org#131388 (status was 'open')

Searchable as RT131388$

[p5pRT]

From @jhi

Something changed in g++ 6 as compared with gcc 6 or g++ 5​: the handling
of subnormal (denormal) floating point values changed so that the Perl
core code parsing (and trying to generate) subnormal numbers does the
"flush-to-zero" instead of retaining the subnormal values.

t/op/sprintf2.t​:
...
ok 1491 - subnormal 3e-324 %.1a 0x1.0p-1074 got 0x1.0p-1074
not ok 1492 - subnormal 0x1.fffffffffffffp-1022 %a
0x1.fffffffffffffp-1022 got 0x0p+0
not ok 1493 - subnormal 0x0.fffffffffffffp-1022 %a
0x1.ffffffffffffep-1023 got 0x0p+0
not ok 1494 - subnormal 0x0.7ffffffffffffp-1022 %a
0x1.ffffffffffffcp-1024 got 0x0p+0
not ok 1495 - subnormal 0x0.3ffffffffffffp-1022 %a
0x1.ffffffffffff8p-1025 got 0x0p+0
not ok 1496 - subnormal 0x0.1ffffffffffffp-1022 %a
0x1.ffffffffffffp-1026 got 0x0p+0
not ok 1497 - subnormal 0x0.0ffffffffffffp-1022 %a
0x1.fffffffffffep-1027 got 0x0p+0
ok 1498
...

The problem seems to be the side of
parsing the hexfp literals, so toke.c Perl_scan_num(), not on the
sv.c sprintf side.

$ ./miniperl.gcc -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
ffffffffffff1f00
$ ./miniperl.g++ -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
0000000000000000

The behavioural change has been confirmed with both in SUSE Linux (g++
(SUSE Linux) 6.3.1 20170202 [gcc-6-branch revision 245119])
and in OS X (with gcc version 6.3.0 (MacPorts gcc6 6.3.0_2)

The discussion in the thread "C++ / G++" in
https://www.nntp.perl.org/group/perl.perl5.porters/2017/05/msg244361.html
or https://marc.info/?l=perl5-porters&m=149475718717687&w=2

One strange thing noticed that if configured with longdoubles, the g++ 6
build does work correctly​:
https://www.nntp.perl.org/group/perl.perl5.porters/2017/05/msg244411.html
or https://marc.info/?l=perl5-porters&m=149495399107618&w=2

[p5pRT]

From @jhi

$ ./miniperl.gcc -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
ffffffffffff1f00
$ ./miniperl.g++ -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
0000000000000000

... and the problem is only with the hexadecimal floating point literals​:

./miniperl.gcc -wle 'print 0x1.fffffffffffffp-1022'
4.4501477170144e-308
./miniperl.g++ -wle 'printf "%a\n", 4.4501477170144e-308'
0x1.ffffffffffff5p-1022

[p5pRT]

From @jkeenan

On Sun, 28 May 2017 06​:03​:56 GMT, jhi wrote​:

Something changed in g++ 6 as compared with gcc 6 or g++ 5​: the handling
of subnormal (denormal) floating point values changed so that the Perl
core code parsing (and trying to generate) subnormal numbers does the
"flush-to-zero" instead of retaining the subnormal values.

t/op/sprintf2.t​:
...
ok 1491 - subnormal 3e-324 %.1a 0x1.0p-1074 got 0x1.0p-1074
not ok 1492 - subnormal 0x1.fffffffffffffp-1022 %a
0x1.fffffffffffffp-1022 got 0x0p+0
not ok 1493 - subnormal 0x0.fffffffffffffp-1022 %a
0x1.ffffffffffffep-1023 got 0x0p+0
not ok 1494 - subnormal 0x0.7ffffffffffffp-1022 %a
0x1.ffffffffffffcp-1024 got 0x0p+0
not ok 1495 - subnormal 0x0.3ffffffffffffp-1022 %a
0x1.ffffffffffff8p-1025 got 0x0p+0
not ok 1496 - subnormal 0x0.1ffffffffffffp-1022 %a
0x1.ffffffffffffp-1026 got 0x0p+0
not ok 1497 - subnormal 0x0.0ffffffffffffp-1022 %a
0x1.fffffffffffep-1027 got 0x0p+0
ok 1498
...

The problem seems to be the side of
parsing the hexfp literals, so toke.c Perl_scan_num(), not on the
sv.c sprintf side.

$ ./miniperl.gcc -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
ffffffffffff1f00
$ ./miniperl.g++ -wle 'print unpack("H*",pack("d",0x1.fffffffffffffp-1022))'
0000000000000000

The behavioural change has been confirmed with both in SUSE Linux (g++
(SUSE Linux) 6.3.1 20170202 [gcc-6-branch revision 245119])
and in OS X (with gcc version 6.3.0 (MacPorts gcc6 6.3.0_2)

The discussion in the thread "C++ / G++" in
https://www.nntp.perl.org/group/perl.perl5.porters/2017/05/msg244361.html
or https://marc.info/?l=perl5-porters&m=149475718717687&w=2

One strange thing noticed that if configured with longdoubles, the g++ 6
build does work correctly​:
https://www.nntp.perl.org/group/perl.perl5.porters/2017/05/msg244411.html
or https://marc.info/?l=perl5-porters&m=149495399107618&w=2

Jarkko, could you supply (at least) the Configure args you used to build the perl used in these tests? Or preferably attach the output of "perl -V"?

We should try to reproduce with still newer versions of g++.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jhi

build does work correctly​:
https://www.nntp.perl.org/group/perl.perl5.porters/2017/05/msg244411.html
or https://marc.info/?l=perl5-porters&m=149495399107618&w=2

Jarkko, could you supply (at least) the Configure args you used to build the perl used in these tests? Or preferably attach the output of "perl -V"?

Uh, sorry, I am pretty certain I do no more have the setup anywhere, and
my builds are ephemeral. But if I go by my usual reflexes the
invocation of Configure has been "sh Configure -des -Dcc=g++".

We should try to reproduce with still newer versions of g++.

Thank you very much.

[p5pRT]

From @sisyphus

On Mon, 03 Sep 2018 10​:33​:33 -0700, jkeenan wrote​:

On Sun, 28 May 2017 06​:03​:56 GMT, jhi wrote​:

The problem seems to be the side of
parsing the hexfp literals, so toke.c Perl_scan_num(), not on the
sv.c sprintf side.

I agree.
I certainly haven't been able to reproduce the behaviour in any C scripts compiled using g++.

We should try to reproduce with still newer versions of g++.

I see the same with g++ 7.3.0 on Ubuntu-18.04.
And it's happening on 5.28.0 and 5.29.2, too, though the problem has been hidden away (via TODO) since sometime prior to the release of 5.28.0 - so that the failures are no longer reported by 'make test'.

Comments in op/sprintf2.t claim that "pow(2.0, -1074) returns 0", but I couldn't find anything to substantiate that claim.

It's not just subnormals that are affected, either​:

~/comp/perl-5.29.2$ ./perl -le 'printf "%a\n", 0x1.0p-1020;'
0x0p+0

though this works as expected when rewritten as​:

~/comp/perl-5.29.2$ ./perl -le 'printf "%a\n", 0x1p-1020;'
0x1p-1020

And it's not limited to g++ builds, either. I found a few subnormal values where a gcc-7.3.0 build of 5.29.2 on the same machine (and a gcc-8.1.0 build of 5.29.2 on Windows 7) were also behaving improperly - eg​:

$ perl -le 'printf "%a\n", 0x1.0p-1074;'
0x0p+0
$ perl -le 'printf "%a\n", 0x1p-1074;'
0x1p-1074

(Same type of behaviour for exponents -1071, -1072, and -1073 on the same 2 machines.)

I Configured the g++ builds with "-des -Dcc=g++" (and additionally with "-Dusedevel" for 5.29.2).

Cheers,
Rob

[p5pRT]

From @iabyn

On Tue, Sep 04, 2018 at 07​:29​:00AM -0700, sisyphus@​cpan.org via RT wrote​:

Comments in op/sprintf2.t claim that "pow(2.0, -1074) returns 0", but I couldn't find anything to substantiate that claim.

The commit which added those comments was mine, although can no longer
remember the details​:

commit 38c84d6
Author​: David Mitchell <davem@​iabyn.com>
AuthorDate​: Tue May 1 15​:28​:49 2018 +0100
Commit​: David Mitchell <davem@​iabyn.com>
CommitDate​: Tue May 1 15​:52​:55 2018 +0100

  sprintf2.t​: mark TODO bad denorm values under g++
 
  Some t/op/sprintf2.t tests were failing under g++. This is due the perl
  toker interpreting very small literal hex floating pointers as 0 rather
  than as a subnormal value.
 
  For example​:
 
  perl -le'print "bad" if 0x1.fffffffffffffp-1022 == 0.0'
 
  This breaks some of the sprintf2.t tests, so mark them TODO them if the
  literal value evaluates to zero.
 
  Note that this is a bug in the toker/g++/glibc rather than sprintf.
 
  The issue is due to the use of pow() in scan_num()​:
 
  under gcc and plain g++, pow(2.0, -1074) returns the smallest denorm
  number; however, under 'g++ -ansi', it returns 0.0.

--
I thought I was wrong once, but I was mistaken.

[p5pRT]

From @sisyphus

On Tue, 04 Sep 2018 12​:48​:48 -0700, davem wrote​:

under gcc and plain g++, pow(2.0, -1074) returns the smallest denorm
number; however, under 'g++ -ansi', it returns 0.0.

Yes, you're right.
I thought I had checked with the ansi switch last night, but apparently I hadn't.

Cheers,
Rob

[p5pRT]

From @sisyphus

On Tue, 04 Sep 2018 12​:48​:48 -0700, davem wrote​:

The issue is due to the use of pow() in scan_num()

The attached patch to current blead fixes the bug for me.

Cheers,
Rob

[p5pRT]

From @sisyphus

0001-toke.c-Cast-I32-to-NV-in-Perl_pow-call.patch

From 9d00f794432f827520b5db60b8a12d897248d1cf Mon Sep 17 00:00:00 2001
From: sisyphus <sisyphus1@optusnet.com.au>
Date: Wed, 5 Sep 2018 21:36:29 +1000
Subject: [PATCH] toke.c - Cast I32 to NV in Perl_pow() call

---
 toke.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/toke.c b/toke.c
index 24e614fd50..191a1ce8b9 100644
--- a/toke.c
+++ b/toke.c
@@ -11221,7 +11221,7 @@ Perl_scan_num(pTHX_ const char *start, YYSTYPE* lvalp)
 #ifdef HEXFP_UQUAD
                         hexfp_exp -= hexfp_frac_bits;
 #endif
-                        hexfp_mult = Perl_pow(2.0, hexfp_exp);
+                        hexfp_mult = Perl_pow(2.0, (NV)hexfp_exp);
                         hexfp = TRUE;
                         goto decimal;
                     }
-- 
2.17.1

[p5pRT]

From @jkeenan

On Wed, 05 Sep 2018 12​:20​:20 GMT, sisyphus@​cpan.org wrote​:

On Tue, 04 Sep 2018 12​:48​:48 -0700, davem wrote​:

The issue is due to the use of pow() in scan_num()

The attached patch to current blead fixes the bug for me.

Cheers,
Rob

Created branch for smoke-testing​:

smoke-me/jkeenan/sisyphus/131388-subnormals

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @sisyphus

On Wed, 05 Sep 2018 14​:17​:46 -0700, jkeenan wrote​:

Created branch for smoke-testing​:

smoke-me/jkeenan/sisyphus/131388-subnormals

If the patch is successful, we will also need to undo Dave's commit (38c84d6) to sprintf2.t that made the failing tests "TODO".

In the one report currently available on the new smoke-me branch, I see the line "Test todo-passed​:" with no entries following it - so I'm guessing that will list any TODOs that failed.

Are g++ builds being smoked ?

Cheers,
Rob

[p5pRT]

From @jkeenan

On Wed, 05 Sep 2018 23​:47​:32 GMT, sisyphus@​cpan.org wrote​:

On Wed, 05 Sep 2018 14​:17​:46 -0700, jkeenan wrote​:

Created branch for smoke-testing​:

smoke-me/jkeenan/sisyphus/131388-subnormals

If the patch is successful, we will also need to undo Dave's commit
(38c84d6) to sprintf2.t that made the
failing tests "TODO".

Just now I reverted that commit and pushed it back to origin in same branch.

In the one report currently available on the new smoke-me branch, I
see the line "Test todo-passed​:" with no entries following it - so I'm
guessing that will list any TODOs that failed.

Are g++ builds being smoked ?

FBOW, there is no centralized schema/matrix of platforms/C-compilers being smoked. All smoke testing rigs are maintained by individual volunteers; none (any longer) by perl.org or perl5-porters. So it depends on what individual testers have had the time and energe to set up.

My recommendation​:

1. Based on the discussion in this RT and in any other relevant places on list, make a list of platform/C-compiler combinations which need to be tested to thoroughly exercise the branch.

2. After you've made that list -- not before, which would introduce a potential bias -- go to http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Fsisyphus%2F131388-subnormals and review the results.

3. Then make a list of platform/C-compiler combos which remain to be tested. Post that back in this RT. We'll see what we can do to accommodate you and the needs of the RT.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

On Thu, 06 Sep 2018 02​:20​:46 GMT, jkeenan wrote​:

On Wed, 05 Sep 2018 23​:47​:32 GMT, sisyphus@​cpan.org wrote​:

On Wed, 05 Sep 2018 14​:17​:46 -0700, jkeenan wrote​:

Created branch for smoke-testing​:

smoke-me/jkeenan/sisyphus/131388-subnormals

If the patch is successful, we will also need to undo Dave's commit
(38c84d6) to sprintf2.t that made
the
failing tests "TODO".

Just now I reverted that commit and pushed it back to origin in same
branch.

In the one report currently available on the new smoke-me branch, I
see the line "Test todo-passed​:" with no entries following it - so
I'm
guessing that will list any TODOs that failed.

Are g++ builds being smoked ?

FBOW, there is no centralized schema/matrix of platforms/C-compilers
being smoked. All smoke testing rigs are maintained by individual
volunteers; none (any longer) by perl.org or perl5-porters. So it
depends on what individual testers have had the time and energe to set
up.

My recommendation​:

1. Based on the discussion in this RT and in any other relevant places
on list, make a list of platform/C-compiler combinations which need to
be tested to thoroughly exercise the branch.

2. After you've made that list -- not before, which would introduce a
potential bias -- go to http​://perl.develop-help.com/?b=smoke-
me%2Fjkeenan%2Fsisyphus%2F131388-subnormals and review the results.

3. Then make a list of platform/C-compiler combos which remain to be
tested. Post that back in this RT. We'll see what we can do to
accommodate you and the needs of the RT.

Thank you very much.

One other consideration​: Your patch added no new tests, though we just reverted the TODO-ing of some older tests. It may be the case -- I haven't really followed the details of this RT that closely -- that to fully resolve the problem we need to write *new* tests. If so, we should add them to this branch and do additional smoke testing.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jhi

Are g++ builds being smoked ?

FBOW, there is no centralized schema/matrix of platforms/C-compilers being smoked. All smoke testing rigs are maintained by individual volunteers; none (any longer) by perl.org or perl5-porters. So it depends on what individual testers have had the time and energe to set up.

My recommendation​:

1. Based on the discussion in this RT and in any other relevant places on list, make a list of platform/C-compiler combinations which need to be tested to thoroughly exercise the branch.

*All* the combinations is of course far too large a number, sadly.
Also, I don't think this discussion should be limited to this one
or any one single branch. In general, I think doing first some varying
on the platform, and then some varying on the compiler, would be good.

The basic boring combination is gcc (or clang) on x86-64 on linux
(and therefore glibc). Anything beyond that would be very useful.

Some degrees to consider​:

- x86-32
- non-x86
- bigendian
- some CPU with stricter alignment than x86

- gcc
- clang
- non-gcc-compatible-cc
- C++ compiler
- cross-compiler

- non-Linux
- non-UNIX
- non-UNIX-other-than-windows

- long double
- threads