Constant strings representing a number can BECOME numbers

[p5pRT]

Migrated from rt.perl.org#20661 (status was 'resolved')

Searchable as RT20661$

[p5pRT]

From perl-5.8.0@ton.iguana.be

Created by perl-5.8.0@ton.iguana.be

perl -wle 'sub fun { print+(shift() & "+0") eq "0" ? "yes" : "no"} fun("waf"); fun(0); fun("waf")'
perl -wle 'sub fun { my $str = shift; print "$str​: ", ($str & "+0") eq "0" ? "numeric" : "string"} fun("waf"); fun(0); fun("waf")'
waf​: string
0​: numeric
Argument "waf" isn't numeric in bitwise and (&) at -e line 1.
waf​: numeric

This was code attempting to determine if a passed argument was "really"
numeric at the perl level or not.

However, it seems that once "+0" has interacted in the & with a number, it
starts to behave like the number 0, so from then on on strings you get the
warning and and everything looks like a number.

While this is "normal" behaviour for variables, I don't think that
constant strings like "+0" should do that.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl v5.8.0:

Configured by ton at Tue Nov 12 01:56:18 CET 2002.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.19, archname=i686-linux-thread-multi-64int-ld
    uname='linux quasar 2.4.19 #5 wed oct 2 02:34:25 cest 2002 i686 unknown '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=undef uselongdouble=define
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -fomit-frame-pointer',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='2.95.3 20010315 (release)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='long double', nvsize=12, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lposix -lcrypt -lutil
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.0:
    /usr/lib/perl5/5.8.0/i686-linux-thread-multi-64int-ld
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i686-linux-thread-multi-64int-ld
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    .


Environment for perl v5.8.0:
    HOME=/home/ton
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/ton/bin.Linux:/home/ton/bin:/home/ton/bin.SampleSetup:/usr/local/bin:/usr/local/sbin:/usr/local/jre/bin:/home/oracle/product/9.0.1/bin:/usr/local/ar/bin:/usr/games/bin:/usr/X11R6/bin:/usr/share/bin:/usr/bin:/usr/sbin:/bin:/sbin:.
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @hvds

"perl-5.8.0@​ton.iguana.be (via RT)" <perlbug-followup@​perl.org> wrote​:
:perl -wle 'sub fun { print+(shift() & "+0") eq "0" ? "yes" : "no"} fun("waf"); fun(0); fun("waf")'
:perl -wle 'sub fun { my $str = shift; print "$str​: ", ($str & "+0") eq "0" ? "numeric" : "string"} fun("waf"); fun(0); fun("waf")'
:waf​: string
:0​: numeric
:Argument "waf" isn't numeric in bitwise and (&) at -e line 1.
:waf​: numeric
:
:This was code attempting to determine if a passed argument was "really"
:numeric at the perl level or not.
:
:However, it seems that once "+0" has interacted in the & with a number, it
:starts to behave like the number 0, so from then on on strings you get the
:warning and and everything looks like a number.
:
:While this is "normal" behaviour for variables, I don't think that
:constant strings like "+0" should do that.

Hmm, tricky I think. In most cases, caching conversion results for
constants is a desirable thing to do.

In this case, the easiest way to avoid it is not to be a constant​:
  sub fun { my $z = "+0"; my $str = shift; print "$str​: ", ($str & $z) eq "0" ? "numeric" :"string"}

What else is broken by upgrading constants in this way?

Hugo

[p5pRT]

From perl5-porters@ton.iguana.be

In article <200302120433.h1C4XdX03020@​crypt.compulink.co.uk>,
  hv@​crypt.org writes​:

"perl-5.8.0@​ton.iguana.be (via RT)" <perlbug-followup@​perl.org> wrote​:
​:perl -wle 'sub fun { print+(shift() & "+0") eq "0" ? "yes" : "no"} fun("waf"); fun(0); fun("waf")'
​:perl -wle 'sub fun { my $str = shift; print "$str​: ", ($str & "+0") eq "0" ? "numeric" : "string"} fun("waf"); fun(0); fun("waf")'
​:waf​: string
​:0​: numeric
​:Argument "waf" isn't numeric in bitwise and (&) at -e line 1.
​:waf​: numeric
​:
​:This was code attempting to determine if a passed argument was "really"
​:numeric at the perl level or not.
​:
​:However, it seems that once "+0" has interacted in the & with a number, it
​:starts to behave like the number 0, so from then on on strings you get the
​:warning and and everything looks like a number.
​:
​:While this is "normal" behaviour for variables, I don't think that
​:constant strings like "+0" should do that.

Hmm, tricky I think. In most cases, caching conversion results for
constants is a desirable thing to do.

Is it really ? How often would somebody write e.g.

$x += "1" ?

In this case, the easiest way to avoid it is not to be a constant​:
sub fun { my $z = "+0"; my $str = shift; print "$str​: ", ($str & $z) eq "0" ? "numeric" :"string"}

What else is broken by upgrading constants in this way?

It probably only matters in the cases where the behaviour of string
and number differ, which is only the bitops I think. Since these are known
for their iffy behaviour, how about just doing the constant upgrade in
general, *except* when involved in a bitop ? If someone writes
$var & "0", he probably *cares*.

Mmm, can the constant conversion actually be done at compile time,
since it's already known there in what kind of operation the constant is
involved ?

[p5pRT]

From @cpansprout

I believed the attached patch is self-explanatory.

[p5pRT]

From @cpansprout

Inline Patch

diff -Nup blead/pp.c blead-20661-str-becomenig-num/pp.c
--- blead/pp.c	2010-07-28 03:15:10.000000000 -0700
+++ blead-20661-str-becomenig-num/pp.c	2010-08-01 14:18:56.000000000 -0700
@@ -2387,6 +2387,10 @@ PP(pp_bit_and)
     {
       dPOPTOPssrl;
       if (SvNIOKp(left) || SvNIOKp(right)) {
+	const bool left_must_not_turn_into_a_number
+	 = cBINOP->op_first->op_type == OP_CONST && !SvNIOKp(left);
+	const bool right_must_not_turn_into_a_number
+	 = cBINOP->op_last->op_type == OP_CONST && !SvNIOKp(right);
 	if (PL_op->op_private & HINT_INTEGER) {
 	  const IV i = SvIV_nomg(left) & SvIV_nomg(right);
 	  SETi(i);
@@ -2395,6 +2399,8 @@ PP(pp_bit_and)
 	  const UV u = SvUV_nomg(left) & SvUV_nomg(right);
 	  SETu(u);
 	}
+	if (left_must_not_turn_into_a_number)  SvNIOK_off(left);
+	if (right_must_not_turn_into_a_number) SvNIOK_off(right);
       }
       else {
 	do_vop(PL_op->op_type, TARG, left, right);
@@ -2413,6 +2419,10 @@ PP(pp_bit_or)
     {
       dPOPTOPssrl;
       if (SvNIOKp(left) || SvNIOKp(right)) {
+	const bool left_must_not_turn_into_a_number
+	 = cBINOP->op_first->op_type == OP_CONST && !SvNIOKp(left);
+	const bool right_must_not_turn_into_a_number
+	 = cBINOP->op_last->op_type == OP_CONST && !SvNIOKp(right);
 	if (PL_op->op_private & HINT_INTEGER) {
 	  const IV l = (USE_LEFT(left) ? SvIV_nomg(left) : 0);
 	  const IV r = SvIV_nomg(right);
@@ -2425,6 +2435,8 @@ PP(pp_bit_or)
 	  const UV result = op_type == OP_BIT_OR ? (l | r) : (l ^ r);
 	  SETu(result);
 	}
+	if (left_must_not_turn_into_a_number)  SvNIOK_off(left);
+	if (right_must_not_turn_into_a_number) SvNIOK_off(right);
       }
       else {
 	do_vop(op_type, TARG, left, right);
diff -Nurp blead/t/op/bop.t blead-20661-str-becomenig-num/t/op/bop.t
--- blead/t/op/bop.t	2009-11-19 08:51:40.000000000 -0800
+++ blead-20661-str-becomenig-num/t/op/bop.t	2010-08-01 14:06:50.000000000 -0700
@@ -15,7 +15,7 @@ BEGIN {
 # If you find tests are failing, please try adding names to tests to track
 # down where the failure is, and supply your new names as a patch.
 # (Just-in-time test naming)
-plan tests => 161 + (10*13*2) + 4;
+plan tests => 170 + (10*13*2) + 4;
 
 # numerics
 ok ((0xdead & 0xbeef) == 0x9ead);
@@ -63,6 +63,20 @@ is (($foo | $bar), ($Aoz x 75 . $zap));
 # ^ does not truncate
 is (($foo ^ $bar), ($Axz x 75 . $zap));
 
+# string constants
+sub _and($) { $_[0] & "+0" }
+sub _oar($) { $_[0] | "+0" }
+sub _xor($) { $_[0] ^ "+0" }
+is _and "waf", '# ',  'str var & const str'; # These three
+is _and  0,    '0',   'num var & const str';    # are from
+is _and "waf", '# ',  'str var & const str again'; # [perl #20661]
+is _oar "yit", '{yt', 'str var | const str';
+is _oar  0,    '0',   'num var | const str';
+is _oar "yit", '{yt', 'str var | const str again';
+is _xor "yit", 'RYt', 'str var ^ const str';
+is _xor  0,    '0',   'num var ^ const str';
+is _xor "yit", 'RYt', 'str var ^ const str again';
+
 #
 is ("ok \xFF\xFF\n" & "ok 19\n", "ok 19\n");
 is ("ok 20\n" | "ok \0\0\n", "ok 20\n");

[p5pRT]

From @cpansprout

On Aug 1, 2010, at 2​:22 PM, Father Chrysostomos wrote​:

I believed the attached patch is self-explanatory.
<open_nKvdUEYU.txt>

Someone pointed out to me that it would likely be reviewed more quickly if I provided a commit message, so here it is​:

This patch solves the problem of $x & "+0" not treating the RHS as a string if, on a previous invocation, the LHS happened to be a number (similarly with the other bitwise ops, too). This patch takes the conservative approach of fixing *just* those cases that have explicit quotation marks in the source code, which are clearly broken (and also ‘use constant’-style string constants, which are indistinguishable). (Read-only variables are slightly controversial still, and my patch does not affect those.)

It does this by making the appropriate pp_ funcitons look at the op tree to see whether either operand is a constant during a numeric bitwise operation. If it is, and it is not numeric, it turns off the numericness (numericality?) before returning.

[p5pRT]

From @cpansprout

On Wed Feb 12 14​:13​:32 2003, perl5-porters@​ton.iguana.be wrote​:

It probably only matters in the cases where the behaviour of string
and number differ, which is only the bitops I think. Since these are
known
for their iffy behaviour, how about just doing the constant upgrade in
general, *except* when involved in a bitop ? If someone writes
$var & "0", he probably *cares*.

Commit b20c4ee stops bitops from coercing read-only arguments.

[p5pRT]

@cpansprout - Status changed from 'open' to 'resolved'