printf %a mishandles subnormals

[p5pRT]

Migrated from rt.perl.org#128843 (status was 'resolved')

Searchable as RT128843$

[p5pRT]

From zefram@fysh.org

Created by zefram@fysh.org

$ perl -lwe 'printf "%a\n", 3e-320'
0x1.00000000017b8p-1062

This output is numerically incorrect. This happens for any subnormal
floating-point value. The output is a mixture of two ways of correctly
describing the value​: it is 0x0.00000000017b8p-1022 (displaying
the physical structure of a subnormal) or 0x1.7b8p-1062 (consistent
scientific style).

$ perl -MData​::Float=float_hex -lwe 'print float_hex(3e-320, {subnormal_strategy=>$_}) for qw(SUBNORMAL NORMAL)'
+0x0.00000000017b8p-1022
+0x1.7b80000000000p-1062

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.24.0:

Configured by zefram at Mon May  9 19:42:55 BST 2016.

Summary of my perl5 (revision 5 version 24 subversion 0) configuration:
   
  Platform:
    osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux-thread-multi
    uname='linux barba.rous.org 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 gnulinux '
    config_args='-des -Dprefix=/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52 -Duselargefiles -Dusethreads -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dusedevel -Uversiononly -Ui_db'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion='', gccversion='4.9.2', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lpthread -lnsl -ldb -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.19.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.19'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'



@INC for perl 5.24.0:
    /home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/site_perl/5.24.0/x86_64-linux-thread-multi
    /home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/site_perl/5.24.0
    /home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0/x86_64-linux-thread-multi
    /home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0
    .


Environment for perl 5.24.0:
    HOME=/home/zefram
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/bin:/home/zefram/usr/perl/util:/home/zefram/pub/x86_64-unknown-linux-gnu/bin:/home/zefram/pub/common/bin:/usr/bin:/bin:/usr/local/bin:/usr/games
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

[p5pRT]

From @tonycoz

On Thu Aug 04 13​:51​:17 2016, zefram@​fysh.org wrote​:

$ perl -lwe 'printf "%a\n", 3e-320'
0x1.00000000017b8p-1062

This output is numerically incorrect. This happens for any subnormal
floating-point value. The output is a mixture of two ways of
correctly
describing the value​: it is 0x0.00000000017b8p-1022 (displaying
the physical structure of a subnormal) or 0x1.7b8p-1062 (consistent
scientific style).

The handling of the implicit one bit is incorrect too.

$ perl -MData​::Float=float_hex -lwe 'print float_hex(3e-320,
{subnormal_strategy=>$_}) for qw(SUBNORMAL NORMAL)'
+0x0.00000000017b8p-1022
+0x1.7b80000000000p-1062

It looks like subnormals simply weren't implemented​:

  /* XXX Inf/NaN/denormal handling in the HEXTRACT_IMPLICIT_BIT,
  * and elsewhere. */

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jhi

http​://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e

[p5pRT]

From zefram@fysh.org

Jarkko Hietaniemi via RT wrote​:

http​://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e

When I first reviewed this, before Jarkko merged it to blead, I thought
it fixed the problem, and declared it good. Actually it's not correct​:
it's changed the problem into a different form. Sorry for not spotting
it first time.

With IEEE double​:

$ perl -lwe 'printf "%a\n", 0x1.fffffffffffffp-1022'
0x1.fffffffffffffp-1022
$ perl -lwe 'printf "%a\n", 0x0.fffffffffffffp-1022'
0x1.fffffffffffffp-1023
$ perl -lwe 'printf "%a\n", 0x0.7ffffffffffffp-1022'
0x1.7ffffffffffffp-1024
$ perl -lwe 'printf "%a\n", 0x0.3ffffffffffffp-1022'
0x1.3ffffffffffffp-1025
$ perl -lwe 'printf "%a\n", 0x0.1ffffffffffffp-1022'
0x1.1ffffffffffffp-1026
$ perl -lwe 'printf "%a\n", 0x0.0ffffffffffffp-1022'
0x1.fffffffffffep-1027

The first one here (which is normalised) is OK, but the subnormals
have gone awry. The second should be "0x1.ffffffffffffep-1023",
the third should be "0x1.ffffffffffffcp-1024", the fourth
should be "0x1.ffffffffffff8p-1025", and the fifth should be
"0x1.ffffffffffffp-1026". The sixth output, also subnormal, is correct.
Smaller subnormals of the same pattern are all correct, down to the
smallest positive value.

All of the incorrect ones have the pattern of showing a normalised-style
integer part and exponent but the fractional part from subnormal style.
That's precisely what was wrong originally for all subnormals.

As far as I can see, the problem arises with any magnitude in the range
[2**-1026, 2**-1022), and not for anything outside that range. I expect
that the reason for exactly four binary orders of magnitude being affected
is down to that being a single order of magnitude of the output radix.

With x86 80-bit long double, the behaviour is stranger​:

$ perl -lwe 'printf "%a\n", 0x1.fffffffffffffffep0 * 2**-16382'
Hexadecimal float​: mantissa overflow at -e line 1.
0xf.fffffffffffffffp-16385
$ perl -lwe 'printf "%a\n", 0x0.fffffffffffffffep0 * 2**-16382'
0x7.fffffffffffffffp-16386
$ perl -lwe 'printf "%a\n", 0x0.7ffffffffffffffep0 * 2**-16382'
0x3.fffffffffffffffp-16387
$ perl -lwe 'printf "%a\n", 0x0.3ffffffffffffffep0 * 2**-16382'
0x1.fffffffffffffffp-16388
$ perl -lwe 'printf "%a\n", 0x0.1ffffffffffffffep0 * 2**-16382'
0x0.fffffffffffffffp-16389
$ perl -lwe 'printf "%a\n", 0x0.0ffffffffffffffep0 * 2**-16382'
0x0.7ffffffffffffffp-16390
$ perl -lwe 'printf "%a\n", 0x0.07fffffffffffffep0 * 2**-16382'
0x0.3ffffffffffffffp-16391
$ perl -lwe 'printf "%a\n", 0x0.03fffffffffffffep0 * 2**-16382'
0x0.1ffffffffffffffp-16392
$ perl -lwe 'printf "%a\n", 0x0.01fffffffffffffep0 * 2**-16382'
0x0.0ffffffffffffffp-16393
$ perl -lwe 'printf "%a\n", 0x0.00fffffffffffffep0 * 2**-16382'
0x0.07fffffffffffffp-16394

Subnormals here are consistently showing the integer and fractional
part from subnormal style but the exponent of normalised style. This is
different from the manifestation that was seen with IEEE double. 5.24.0
shows the same behaviour as blead here. Thus the manifestation of the
subnormal problem was always different depending on the NV format, and
Jarkko's patch hasn't affected the behaviour on x86 80-bit long double.

Things get even weirder for certain subnormals in this format​:

$ perl -lwe 'printf "%a\n", 0x0.1000000000000000p0 * 2**-16382'
0x08p-16389
$ perl -lwe 'printf "%a\n", 0x0.1e00000000000000p0 * 2**-16382'
0x0fp-16389
$ perl -lwe 'printf "%a\n", 0x0.0000000000000040p0 * 2**-16382'
0x000000000000002p-16443
$ perl -lwe 'printf "%a\n", 0x0.0000000000000020p0 * 2**-16382'
0x000000000000001p-16444
$ perl -lwe 'printf "%a\n", 0x0.000000000000001ep0 * 2**-16382'
0x000000000000000fp-16445

Where only one output hex digit is non-zero, in addition to the initial
subnormal bug the output is missing its radix point. This too hasn't
changed between 5.24.0 and blead. This is presumably because if the
values were output correctly normalised these ones would not require a
radix point.

-zefram

[p5pRT]

From @jkeenan

On Sat Aug 13 16​:19​:16 2016, zefram@​fysh.org wrote​:

Jarkko Hietaniemi via RT wrote​:

http​://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e

When I first reviewed this, before Jarkko merged it to blead, I
thought
it fixed the problem, and declared it good. Actually it's not
correct​:
it's changed the problem into a different form. Sorry for not
spotting
it first time.

I've been following the discussion in this RT, as well as noting the filing of 5 other bug reports related to 'printf'. Those other reports are​:

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128888
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128893
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128889
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128890
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128909

(Each of the above is still in New status.)

I have to admit I'm confused as to where the discussion in going. In particular, I'm concerned about the fact that no one has cited any standards for the behavior of 'printf' that we presumably have failed to meet up until now and ought to be meeting going forward. It's not clear on what basis, other than personal knowledge, participants in the discussion are making their claims or making code changes. For those following along at home, this is unsatisfying.

Could we step back from the discussion for a moment and provide some links to standards we ought to be meeting?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From zefram@fysh.org

James E Keenan via RT wrote​:

In particular, I'm concerned about the fact that no one has cited any
standards for the behavior of 'printf' that we presumably have failed
to meet up until now and ought to be meeting going forward.

I've based my expectations on the C standard, to the extent that it's
not overridden by deliberate differences in Perl. I've mentioned it in
a couple of the bug reports. The actual standard isn't freely available
(in either sense of "free"), but a late draft of C99 is available at
<http​://std.dkuug.dk/JTC1/SC22/WG14/www/docs/n843.pdf>. In that document,
the description of printf-style formatting starts on page 271, section
7.19.6.1 "The fprintf function".

-zefram

[p5pRT]

From @jhi

I've been following the discussion in this RT, as well as noting the

filing of 5 other bug reports related to 'printf'. Those other
reports are​:

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128888
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128893
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128889
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128890
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128909

(Each of the above is still in New status.)

I have to admit I'm confused as to where the discussion in going. In
particular, I'm concerned about the fact that no one has cited any
standards for the behavior of 'printf' that we presumably have failed
to meet up until now and ought to be meeting going forward. It's not
clear on what basis, other than personal knowledge, participants in
the discussion are making their claims or making code changes. For
those following along at home, this is unsatisfying.

Could we step back from the discussion for a moment and provide some
links to standards we ought to be meeting?

Yeah, sorry. I see Zefram already pointed out some of the relevant docs.

In addition to the C99 (the first one in which the hexfp literals or the %a
came along), I'm following what the majority of platforms seems to have implemented.

There often seems to be some differences in the interpretation of the standards,
but if two our of three independent implementations seem to agree on a behavior,
I'm happy to follow it, unless of course the behavior clearly disagrees with some standard.

[p5pRT]

From @jhi

When I first reviewed this, before Jarkko merged it to blead, I
thought
it fixed the problem, and declared it good. Actually it's not
correct​:
it's changed the problem into a different form. Sorry for not
spotting
it first time.

Hrm. I'll try to take a look again tonight.

[p5pRT]

From @jhi

Zefram, before I go off and try to fix the new findings, could you please check t/op/sprintf2.t
and t/op/hexfp.t as of blead 7301378, and see that there are no accidentally wrong behaviors
in the newly added tests, especially the subnormals ones?

[p5pRT]

From zefram@fysh.org

Jarkko Hietaniemi via RT wrote​:

Zefram, before I go off and try to fix the new findings, could you
please check t/op/sprintf2.t and t/op/hexfp.t as of blead 7301378,

sprintf2.t for double-double​:

  [ '%a', '-1', '-0x0p+0' ],

Expected result should be '-0x1p+0'.

sprintf2.t has a comment about tests for this ticket (#128843) for the
80-bit format, but it doesn't actually have any tests for values that
are subnormal in that format. It only tests 3e-320, which is normal
in 80-bit.

Other than that the %a tests look fine.

-zefram

[p5pRT]

From @jhi

Marking resolved until proven otherwise.

[p5pRT]

@jhi - Status changed from 'open' to 'resolved'