t/re/uniprops02.t failures when NV is "doubledouble" long double

[sisyphus]

For quite some time, t/re/uniprops02.t has been failing some tests whenever I build perl such that the type of NV is the "doubledouble" long double.

The first test that fails is:

not ok 30410 - couldn't compile /\p{Numeric_Value:3.12e-03}/; line 69958: Can't find Unicode property definition "Numeric_Value:3.12e-03" in regex; marked by <-- HERE in m/\p{Numeric_Value:3.12e-03} <-- HERE / at (eval 26922) line 2.

This is in response to the following line of code in lib/unicore/TestProp.pl:

Expect(1, 73684, '\p{Numeric_Value:3.12e-03}', "");

On other systems, that line of the generated TestProp.pl specifies 3.13e-03 instead of 3.12e-03.
(The test passes if the value of 3.13e-03 is used.)

On this "doubledouble" sysyem of mine, the value of 3.12e-03 (or equivalent) is repeated in various other tests throughout TestProp.pl, causing those tests to fail in every instance.
Perhaps this is indicative of a problem with the way that the test values are generated by lib/unicore/mktables ?

But there's another problem, too.
For values of 3.125e-03 (or equivalent), which are also repeated in various tests throughout TestProp.pl, these tests are failing (in the same way) on the "doubledouble" build, but passing on other systems.

Sometimes "doubledoubles" are not assigned correctly (ie can be off by one or two ULPs), but this doesn't seem to be relevant here.
In fact, one value that is off by three ULPs (3.13e-03) is handled correctly, while the one that is assigned correctly (3.125e-03) fails.

I guess it's not critical that all perl bugs related to this very rare type of NV be fixed.
OTOH, perl would be no worse off if it could be fixed.
Please let me know if there's something further I can do to help with this.

Steps to Reproduce

$ perl -le 'use re; my $re = qr[\p{Numeric_Value: 3.12e-03}];'
Can't find Unicode property definition "Numeric_Value: 3.12e-03" in regex; marked by <-- HERE in m/\p{Numeric_Value: 3.12e-03} <-- HERE / at -e line 1.
$ perl -le 'use re; my $re = qr[\p{Numeric_Value: 3.125e-03}];'
Can't find Unicode property definition "Numeric_Value: 3.125e-03" in regex; marked by <-- HERE in m/\p{Numeric_Value: 3.125e-03} <-- HERE / at -e line 1.
$ perl -le 'use re; my $re = qr[\p{Numeric_Value: 0.003125}];'
Can't find Unicode property definition "Numeric_Value: 0.003125" in regex; marked by <-- HERE in m/\p{Numeric_Value: 0.003125} <-- HERE / at -e line 1.

Expected behavior

I'm not very well informed on Unicode matters, but I believe the output of the first one-liner is as expected - the problem there being that it leads to a FAIL when it's run in t/re/uniprops02.t.
IIUC, the second and third one-liners would be expected to produce no output at all.

Perl configuration

$ perl -V
Summary of my perl5 (revision 5 version 35 subversion 11) configuration:
  Commit id: 2caf82775aff58fd4599ee9fb0841a887715c56b
  Platform:
    osname=linux
    osvers=3.2.0-4-powerpc64
    archname=ppc64-linux-ld
    uname='linux debian-sis 3.2.0-4-powerpc64 #1 smp debian 3.2.78-1 ppc64 gnulinux '
    config_args='-des -Duse64bitall -Dcc=gcc -m64 -Dprefix=/home/sisyphus-sis/perl53511-m64-ld -Duselongdouble -Dusedevel -Uversiononly'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=define
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='gcc -m64'
    ccflags ='-std=gnu99 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O1'
    cppflags='-std=gnu99 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion=''
    gccversion='4.6.3'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=87654321
    doublekind=4
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=6
    ivtype='long'
    ivsize=8
    nvtype='long double'
    nvsize=16
    Off_t='off_t'
    lseeksize=8
    alignbytes=16
    prototype=define
  Linker and Libraries:
    ld='gcc -m64'
    ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib/powerpc-linux-gnu /lib/../lib /usr/lib/powerpc-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.13.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.13'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O1 -L/usr/local/lib -fstack-protector'


Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_TIMES
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_DEVEL
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_LONG_DOUBLE
    USE_PERLIO
    USE_PERL_ATOF
  Built under linux
  Compiled at Apr  5 2022 18:11:14
  @INC:
    /home/sisyphus-sis/perl53511-m64-ld/lib/site_perl/5.35.11/ppc64-linux-ld
    /home/sisyphus-sis/perl53511-m64-ld/lib/site_perl/5.35.11
    /home/sisyphus-sis/perl53511-m64-ld/lib/5.35.11/ppc64-linux-ld
    /home/sisyphus-sis/perl53511-m64-ld/lib/5.35.11

[demerphq]

Hi, this needs @khwilliamson to say for sure but I did some digging and I think I can give you some places to look into this yourself.

First, as far as I can tell this related to Unicode's support for fractions (I had no idea), and this relates to 1/320.

perl -le'printf "%.2e\n", 1/320'
3.13e-03

In uni_keywords.h you can find this and similar entries. The table is unordered as it is a 2-level minimal perfect hash table (again I had no idea we had this in core!)

  {     0, 10268,  8052,   3,   8, UNI_NV__1_SLASH_320 }   /* nv=3.13e-03 */,
...
  {     1,  6083,  8052,  13,   8, UNI_NV__1_SLASH_320 }   /* numericvalue=3.13e-03 */,

As far as I can tell there are only a few uses of %e with sprintf in mktables, most of them relate to handling the rational properties. There are two in register_fraction(), and one in the block starting at line 18811, with the comment

                        # If the name is a rational number, add tests for a
                        # non-reduced form, and for a floating point equivalent.

The actual use is at line 18828. There is one more but I dont /think/ it is related.

You mentioned in a prior ticket that there are bugs in sprintf for DoubleDouble, so I guess your problem is there. Maybe you can dig into yourself with this information.

This has been an fascinating education into the work that Karl has done to our Unicode support. So much interesting stuff there, it is quite impressive.

HTH.

[khwilliamson]

On 4/9/2022 8:12 AM, Yves Orton wrote: You mentioned in a prior ticket that there are bugs in sprintf for DoubleDouble, so I guess your problem is there. Maybe you can dig into yourself with this information

Unicode supports rational numbers, hence ones that can be precisely represented as a numerator divided by a denominator. But these aren't likely to be exactly representable as NVs. You can ask for \p{nv=1/320} and get the right answer, but Perl also lets you specify a close-enough NV to mean the same thing. mktables goes to some efforts to calculate what "close enough" is on the platform being compiled on. Apparently it is failing on this case. Could this be the bugs that were referred to?

[sisyphus]

I had spent some time trying to relate errors in assignment to these test failures - without success.
As @demerphq correctly worked out, on this "doubledouble" system, the first failure is a sprintf bug:

$ perl -le 'printf "%.2e\n", 1/320;' 
3.12e-03

However, it's still a puzzle (to me, at least) that \p{Numeric_Value: 3.125e-03} cannot find "Unicode property definition", whereas \p{Numeric_Value: 1/320} apparently can:

$ perl -le 'use re; $re = qr[\p{Numeric_Value: 3.125e-03}];'
Can't find Unicode property definition "Numeric_Value: 3.125e-03" in regex; marked by <-- HERE in m/\p{Numeric_Value: 3.125e-03} <-- HERE / at -e line 1.
$ perl -le 'use re; $re = qr[\p{Numeric_Value: 1/320}];'
$

Both 1/320 and 3.125e-03 are exactly the same doubledouble:

$ perl -le 'print unpack "H*", pack "D", 1/320; print unpack "H*", pack "D", 3.125e-03 ;'
3f6999999999999abc0999999999999a
3f6999999999999abc0999999999999a
$

Cheers,
Rob

[demerphq]

Basically these values are stored in a special hash table. So 1/320 is explicitly stored in the table as 1/320, but also as 3.13e-03, as the rounded "canonical" engineering notation representation of 1/320.

If you look in uni_props.h there is a string blob where the fragments that are part of the keys are stored into a single large blob. If you search that blob you will find 1/320 and 3.13e-03, but you wont find 3.12e-03, nor will you find 3.125e-03. Whether you should find them is a different question that I would have to leave to @khwilliamson to say for sure, but I do not think they should.

What I see when I use gdb to trace is that

./perl -Ilib -le 'my $re = qr[\p{Numeric_Value: 3.125e-03}];'

ends up searching for: numericvalue=3.125e-03 which it does not find, and then it tries to find numericvalue=3.13e-03 (the Numeric_Value is turned into the canonical form of "numericvalue" internally.

The value used in the second look is created at regcomp.c line 25236:

                    canonical = Perl_form(aTHX_ "%.*s%.*" NVef,
                                                equals_pos, lookup_name,
                                                PL_E_FORMAT_PRECISION, value);

So I assume that this is a similar rounding bug as the one that is biting you.

One way to fix these bugs would be to store more rational representations in uni_keywords.h, another would be to fix the rounding in sprintf somehow. It appears that on your system you are rounding down. I think maybe another way would be totally regenerate uni_keywords.h on your system. I think if you delete it and then run perl regen/mk_invlists.pl it should fix itself. You might have to wait a while, I am not sure but I think maybe the generator script is slow.[1]

Considering this is all to work around bugs in sprintf and the use of these properties is rare it might be better to just ignore them, although I can understand if that seems unsatisfactory as it seems unsatisfactory to me too.

[1] It turns out I wrote the generator and had completely forgotten that I did so(!!!!). Now that I have reviewed the code and email backlog I remember it, but I am not sure if I implemented every possible optimization to make the generator faster and I could maybe do something to speed it up.

[demerphq]

I think if you delete it and then run perl regen/mk_invlists.pl it should fix itself

It turns out that this is not true. You need to run it with the latest perl:

$ ./perl -Ilib regen/mk_invlists.pl
No preprocessing, initial blob size 0
Length old blob: 12360 length new blob: 12308, recomputing using new blob
Length old blob: 12308 length new blob: 11874, recomputing using new blob
Length old blob: 11874 length new blob: 11880, keeping old blob
Using preprocessing, initial blob size 35587
Length old blob: 35587 length new blob: 10402, recomputing using new blob
Length old blob: 10571 length new blob: 10586, keeping old blob
Using preprocess-smart blob, length: 10571 (vs 11874)
Changed: uni_keywords.h

Aside: Those last few lines of output are a bit weird to me, I don't know why the length of 10402 magically becomes 10571. I will investigate.

[sisyphus]

Damn ... I need to backtrack and make a correction.
There is no "doubledouble" bug at play in that first failure.
The correct "doubledouble" representation of 1/320, is marginally less than the decimal value 3.125e-03.
It is therefore correct that sprintf "%.2e", 3.125e-03; return 3.12e-03 on that platform.
And I believe it's also correct behaviour that the test therefore fails in the way it does.

I'm not quite sure how I managed to screw that up the first time round. (I think that hastily formulated one-liners might go a long way towards explaining it.)
Correct approach (using Math::MPFR) is to:

1. calculate 1/320 to 2098 bits of precision;
   Note: Using 2098 bits of precision will produce the correct result, irrespective of the value being assigned.
2. assign that 2098-bit value to a doubledouble (some precision is lost in this instance);
3. assign that doubledouble back to another 2098-bit precision mpfr object (no more precision is lost);
4. perform "%.2e" formatting of the value held by that second object, using Rmpfr_printf().
   In all operations, round to nearest with ties to even.

The script:

use strict;
use warnings;

use Math::MPFR qw(:mpfr);

Rmpfr_set_default_prec(2098);

my $obj1 = Math::MPFR->new(1) / 320;

# Let's also check to see if that value
# is equivalent to the value rendered by
# assigning the string '3.125e-03'.
print "1: Equivalent\n" if $obj1 == Math::MPFR->new('3.125e-03');

# Now assign the correct "doubledouble"
# representation of 1/320 to $nv.
# Some precision is lost in this assignment.
my $nv = Rmpfr_get_NV($obj1, MPFR_RNDN);

# Let's also check to see if the value of
# $nv is identical to what perl assigns.
print "2: Equivalent\n" if $nv == 1 / 320;

# Now assign $nv back to another 2098-bit
# precision Math::MPFR object. (No more
# precision is lost in this assignment.)
my $obj2 = Math::MPFR->new($nv);

# Use Rmpfr_printf() instead of perl's
# printf() as Rmpfr_printf() is reliable.
Rmpfr_printf("%.2Re\n", $obj2);

# And finally show that this confirms what
# perl came up with in the first place.
printf("%.2e\n", 1 / 320);

__END__

If nvtype is "doubledouble" long double
this outputs for me:
1: Equivalent
2: Equivalent
3.12e-03
3.12e-03

[demerphq]

Hrm, This suggests to me that canonicalizing to two decimal places of E notation is not appropriate and we should be using 3. At this point I think we need to @khwilliamson to speak up and take over.

[khwilliamson]

@sisyphus try out this branch