Numifying a 0x string evaluates the hex number

[p5pRT]

Migrated from rt.perl.org#134230 (status was 'pending release')

Searchable as RT134230$

[p5pRT]

From @Grinnz

When starting with a string that looks like '0xFF', numifying it should
truncate it to 0 - consistent with 0b and 0 which only have an effect on
bare number parsing. This behavior seems to have changed in 5.30.

5.28​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
0

5.30​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
255

It also seems to have had this behavior way back in 5.6​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
255

-Dan

[p5pRT]

From @Tux

On Thu, 27 Jun 2019 07​:35​:45 -0700, "Dan Book \(via RT\)"
<perlbug-followup@​perl.org> wrote​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'

Interesting

$ perl-all -le 'print "0xFF" + 0'
Running perl-all -le print "0xFF" + 0

=== base/perl5.6.0 5.006 x86_64-linux
255
=== base/perl5.6.1 5.006001 x86_64-linux-perlio
255
=== base/tperl5.6.1 5.006001 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.6.2 5.006002 x86_64-linux-perlio
255
=== base/tperl5.6.2 5.006002 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.6 5.006002 x86_64-linux-perlio
255
=== base/tperl5.6 5.006002 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.8.0 5.008 x86_64-linux
0
=== base/tperl5.8.0 5.008 x86_64-linux-thread-multi-ld
0
=== base/perl5.8.1 5.008001 x86_64-linux
0
=== base/tperl5.8.1 5.008001 x86_64-linux-thread-multi-ld
0
=== base/perl5.8.2 5.008002 x86_64-linux
0

... 0 all the way to

=== base/tperl5.28 5.028001 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.0 5.029000 x86_64-linux
0
=== base/tperl5.29.0 5.029000 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.1 5.029001 x86_64-linux
0
=== base/tperl5.29.1 5.029001 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.2 5.029002 x86_64-linux
255
=== base/tperl5.29.2 5.029002 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.3 5.029003 x86_64-linux
255
=== base/tperl5.29.3 5.029003 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.4 5.029004 x86_64-linux
255
=== base/tperl5.29.4 5.029004 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.5 5.029005 x86_64-linux
255
=== base/tperl5.29.5 5.029005 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.6 5.029006 x86_64-linux
255
=== base/tperl5.29.6 5.029006 x86_64-linux-thread-multi-ld
255
=== base/perl5.29 5.029006 x86_64-linux
255
=== base/tperl5.29 5.029006 x86_64-linux-thread-multi-ld
255

I have more perl's at home

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @sisyphus

I expect this will be due to commit
ce6f496, with which RT 41202 was resolved.
With that commit, we can expect that expressions like "0xFF"+0 and
"0b1001"+0 will numify to the same NV as POSIX​::strtod("0xFF") and
POSIX​::strtod("0b1001") respectively.
Is this a behaviour that should be changed ?

For me, strtod() does not understand the '0b' prefix and simply numifies
such strings to an NV of zero.

C​:\>perl -MPOSIX -le "@​x=POSIX​::strtod('0b11111'); print for @​x;"
0
6

But it does understand the '0x' prefix​:

C​:\>perl -MPOSIX -le "@​x=POSIX​::strtod('0xff'); print for @​x;"
255
0

Cheers,
Rob

On Fri, Jun 28, 2019 at 12​:50 AM H.Merijn Brand <h.m.brand@​xs4all.nl> wrote​:

On Thu, 27 Jun 2019 07​:35​:45 -0700, "Dan Book \(via RT\)"
<perlbug-followup@​perl.org> wrote​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'

Interesting

$ perl-all -le 'print "0xFF" + 0'
Running perl-all -le print "0xFF" + 0

=== base/perl5.6.0 5.006 x86_64-linux
255
=== base/perl5.6.1 5.006001 x86_64-linux-perlio
255
=== base/tperl5.6.1 5.006001 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.6.2 5.006002 x86_64-linux-perlio
255
=== base/tperl5.6.2 5.006002 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.6 5.006002 x86_64-linux-perlio
255
=== base/tperl5.6 5.006002 x86_64-linux-thread-multi-ld-perlio
255
=== base/perl5.8.0 5.008 x86_64-linux
0
=== base/tperl5.8.0 5.008 x86_64-linux-thread-multi-ld
0
=== base/perl5.8.1 5.008001 x86_64-linux
0
=== base/tperl5.8.1 5.008001 x86_64-linux-thread-multi-ld
0
=== base/perl5.8.2 5.008002 x86_64-linux
0

... 0 all the way to

=== base/tperl5.28 5.028001 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.0 5.029000 x86_64-linux
0
=== base/tperl5.29.0 5.029000 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.1 5.029001 x86_64-linux
0
=== base/tperl5.29.1 5.029001 x86_64-linux-thread-multi-ld
0
=== base/perl5.29.2 5.029002 x86_64-linux
255
=== base/tperl5.29.2 5.029002 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.3 5.029003 x86_64-linux
255
=== base/tperl5.29.3 5.029003 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.4 5.029004 x86_64-linux
255
=== base/tperl5.29.4 5.029004 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.5 5.029005 x86_64-linux
255
=== base/tperl5.29.5 5.029005 x86_64-linux-thread-multi-ld
255
=== base/perl5.29.6 5.029006 x86_64-linux
255
=== base/tperl5.29.6 5.029006 x86_64-linux-thread-multi-ld
255
=== base/perl5.29 5.029006 x86_64-linux
255
=== base/tperl5.29 5.029006 x86_64-linux-thread-multi-ld
255

I have more perl's at home

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @sisyphus

On Thu, 27 Jun 2019 18​:14​:38 -0700, sisyphus359@​gmail.com wrote​:
.....

Is this a behaviour that should be changed ?
....

Seems to me that it's maybe a bit silly to be handing hex *integer* strings to strtod() when perl assigns hex integer barewords to IV.

It's ok when the value fits in both the IV and the NV, but when IV-precision exceeds NV-precision we can get results that don't DWIM all that well​:

C​:\> perl -le "print 'wtf' if 0xfffffffffffffff == '0xfffffffffffffc0' + 0;"
wtf

(That's on perl-5.30.0, ivsize and nvsize both == 8.)

It might make better sense if, in such cases, the hex integer string was handed over to srtol/strtoll.
However, strtol("023", NULL, 0) would return the decimal value 19 (same value as the perl bareword 023), but perl has always numified '023' as the decimal value 23, so there's some awkwardness there that would need to be considered.

Cheers,
Rob

[p5pRT]

From @ikegami

Changing this means changing Scalar​::Util​::looks_like_number. Would that
break things?

Also, it could break things that use the regex patterns provided in
perldata to (effectively) recreate looks_like_number.

On Thu, Jun 27, 2019 at 10​:35 AM Dan Book (via RT) <
perlbug-followup@​perl.org> wrote​:

# New Ticket Created by Dan Book
# Please include the string​: [perl #134230]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=134230 >

When starting with a string that looks like '0xFF', numifying it should
truncate it to 0 - consistent with 0b and 0 which only have an effect on
bare number parsing. This behavior seems to have changed in 5.30.

5.28​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
0

5.30​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
255

It also seems to have had this behavior way back in 5.6​:

perl -Mwarnings -e 'print "0xFF" + 0, "\n"'
Argument "0xFF" isn't numeric in addition (+) at -e line 1.
255

-Dan

[p5pRT]

From @sisyphus

On Thu, 27 Jun 2019 22​:34​:32 -0700, ikegami@​adaelis.com wrote​:

Changing this means changing Scalar​::Util​::looks_like_number. Would that
break things?

I agree that looks_like_number()also needs to be thought about.

Perl's current behaviour is to issue the "isn't numeric" warning when "0xff"+0 is evaluated.
However, I doubt that such a warning is valid now that "0xff" is numified to 255.
It was certainly a valid warning when "0xff" numified to 0 ... but I'm thinking that warning should not be emitted for the current behaviour, and nor should looks_like_number("0xff") return FALSE.

Cheers,
Rob

[p5pRT]

From @tonycoz

On Fri, 28 Jun 2019 03​:36​:08 -0700, sisyphus@​cpan.org wrote​:

On Thu, 27 Jun 2019 22​:34​:32 -0700, ikegami@​adaelis.com wrote​:

Changing this means changing Scalar​::Util​::looks_like_number. Would
that
break things?

I agree that looks_like_number()also needs to be thought about.

Perl's current behaviour is to issue the "isn't numeric" warning when
"0xff"+0 is evaluated.
However, I doubt that such a warning is valid now that "0xff" is
numified to 255.
It was certainly a valid warning when "0xff" numified to 0 ... but I'm
thinking that warning should not be emitted for the current behaviour,
and nor should looks_like_number("0xff") return FALSE.

I don't think looks_like_number() needs to change - I think C<"0xff"> needs to return to numerically evaluating as 0, pre the attached.

Tony

[p5pRT]

From @tonycoz

0001-perl-134230-don-t-interpret-0x-0b-when-numifying-str.patch

From e3299a45a90066de1fc387fd0f40cdd81571d3c8 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 20 Aug 2019 15:43:05 +1000
Subject: (perl #134230) don't interpret 0x, 0b when numifying strings

---
 numeric.c  | 9 +++++++++
 t/op/int.t | 5 ++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/numeric.c b/numeric.c
index f5eadc8173..fae2eb3c6d 100644
--- a/numeric.c
+++ b/numeric.c
@@ -1551,6 +1551,15 @@ Perl_my_atof3(pTHX_ const char* orig, NV* value, const STRLEN len)
         if ((endp = S_my_atof_infnan(aTHX_ s, negative, send, value)))
             return endp;
 
+        /* strtold() accepts 0x-prefixed hex and in POSIX implementations,
+           0b-prefixed binary numbers, which is backward incompatible
+        */
+        if ((len == 0 || len >= 2) && *s == '0' &&
+            (isALPHA_FOLD_EQ(s[1], 'x') || isALPHA_FOLD_EQ(s[1], 'b'))) {
+            *value = 0;
+            return (char *)s+1;
+        }
+
         /* If the length is passed in, the input string isn't NUL-terminated,
          * and in it turns out the function below assumes it is; therefore we
          * create a copy and NUL-terminate that */
diff --git a/t/op/int.t b/t/op/int.t
index 7e936da68d..b730ab2672 100644
--- a/t/op/int.t
+++ b/t/op/int.t
@@ -7,7 +7,7 @@ BEGIN {
     require Config;
 }
 
-plan 17;
+plan 19;
 
 # compile time evaluation
 
@@ -83,3 +83,6 @@ SKIP:
         cmp_ok($x, "==", int($x), "check $x == int($x)");
     }
 }
+
+is(1+"0x10", 1, "check string '0x' prefix not treated as hex");
+is(1+"0b10", 1, "check string '0b' prefix not treated as binary");
-- 
2.11.0

[p5pRT]

From @karenetheridge

On Mon, Aug 19, 2019 at 10​:53 PM Tony Cook via RT
<perlbug-followup@​perl.org> wrote​:

On Fri, 28 Jun 2019 03​:36​:08 -0700, sisyphus@​cpan.org wrote​:

On Thu, 27 Jun 2019 22​:34​:32 -0700, ikegami@​adaelis.com wrote​:

Changing this means changing Scalar​::Util​::looks_like_number. Would
that
break things?

I agree that looks_like_number()also needs to be thought about.

Perl's current behaviour is to issue the "isn't numeric" warning when
"0xff"+0 is evaluated.
However, I doubt that such a warning is valid now that "0xff" is
numified to 255.
It was certainly a valid warning when "0xff" numified to 0 ... but I'm
thinking that warning should not be emitted for the current behaviour,
and nor should looks_like_number("0xff") return FALSE.

I don't think looks_like_number() needs to change - I think C<"0xff"> needs to return to numerically evaluating as 0, pre the attached.

Why did this behaviour change? Was it intentional or a side effect?
I have bisected it down to this commit (between the 5.29.1 and 5.29.2 releases)​:

---

commit ce6f496 (HEAD, refs/bisect/bad)
Author​: sisyphus <sisyphus1@​optusnet.com.au>
Date​: 2018-08-01 22​:33​:38 +1000

  PATCH​: [perl #41202] text->float gives wrong answer

  This changes to use Perl_strtod() when available, and that turns out to
  be the key to fixing this bug.

  S_mulexp10() is removed from embed.fnc to avoid repeating the
  complicated prerequisites for defining Perl_strtod(). This works
  because this static function already was defined before use in
  numeric.c, and always called in full form without using a macro.

  James Keenan fixed a file permissions problem originally introduced by
  this commit, but the fix has been squashed into it.

---

[p5pRT]

From @sisyphus

On Mon, Aug 26, 2019 at 5​:23 AM Karen Etheridge <perl@​froods.org> wrote​:

Why did this behaviour change? Was it intentional or a side effect?

An unintended side effect.
We didn't realize that the change had happened until this ticket was raised.

Cheers,
Rob

[p5pRT]

From @tonycoz

On Mon, 19 Aug 2019 22​:53​:04 -0700, tonyc wrote​:

On Fri, 28 Jun 2019 03​:36​:08 -0700, sisyphus@​cpan.org wrote​:

On Thu, 27 Jun 2019 22​:34​:32 -0700, ikegami@​adaelis.com wrote​:

Changing this means changing Scalar​::Util​::looks_like_number. Would
that
break things?

I agree that looks_like_number()also needs to be thought about.

Perl's current behaviour is to issue the "isn't numeric" warning when
"0xff"+0 is evaluated.
However, I doubt that such a warning is valid now that "0xff" is
numified to 255.
It was certainly a valid warning when "0xff" numified to 0 ... but
I'm
thinking that warning should not be emitted for the current
behaviour,
and nor should looks_like_number("0xff") return FALSE.

I don't think looks_like_number() needs to change - I think C<"0xff">
needs to return to numerically evaluating as 0, pre the attached.

Applied as 14d26b4.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'pending release'

[p5pRT]

From @karenetheridge

Excellent!
Could this fix please also be considered for backporting to 5.30-maint?

On Sun, Aug 25, 2019 at 5​:29 PM Tony Cook via RT
<perlbug-followup@​perl.org> wrote​:

On Mon, 19 Aug 2019 22​:53​:04 -0700, tonyc wrote​:

On Fri, 28 Jun 2019 03​:36​:08 -0700, sisyphus@​cpan.org wrote​:

On Thu, 27 Jun 2019 22​:34​:32 -0700, ikegami@​adaelis.com wrote​:

Changing this means changing Scalar​::Util​::looks_like_number. Would
that
break things?

I agree that looks_like_number()also needs to be thought about.

Perl's current behaviour is to issue the "isn't numeric" warning when
"0xff"+0 is evaluated.
However, I doubt that such a warning is valid now that "0xff" is
numified to 255.
It was certainly a valid warning when "0xff" numified to 0 ... but
I'm
thinking that warning should not be emitted for the current
behaviour,
and nor should looks_like_number("0xff") return FALSE.

I don't think looks_like_number() needs to change - I think C<"0xff">
needs to return to numerically evaluating as 0, pre the attached.

Applied as 14d26b4.

Tony

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=134230