overloading 'x' is broken

[p5pRT]

Migrated from rt.perl.org#121827 (status was 'resolved')

Searchable as RT121827$

[p5pRT]

From @salva

Created by @salva

Create the following module​:

------------overx.pm--------------------

package overx;

require overload;
overload->import('x'  => \&_x,
                 '*'  => \&_x,
                 '""' => \&_to_string,
                 '0+' => \&_length);

sub new {
    my ($class, @​a) = @​_;
    my $self = \@​a;
    bless $self, $class;
}

sub _x {
    my $a = shift;
    my $b = shift;
    my $c = [@​$a, @​$b];
    bless $c;
    print STDERR "$a x $b -> $c\n";
    $c;
}

sub _length {
    my $a = shift;
    scalar @​$a;
}

sub _to_string {
    my $a = shift;
    "{".join(', ', @​$a)."}";
}

1;

------------end-of-overx.pm--------------------

and run...

  $ perl -I. -Moverx -MData​::Dumper -e '$a = overx->new(1,'a'); print Dumper [$a, scalar(($a x $a) x $a)]'

I get...

    {1, a} x {1, a} -> {1, a, 1, a}
    {1, a, 1, a} x {1, a} -> {1, a, 1, a, 1, a}
    $VAR1 = bless( [
                     1,
                     'a'
                   ], 'overx' );
    $VAR2 = [
              bless( [
                       1,
                       'a',
                       1,
                       'a',
                       1,
                       'a'
                     ], 'overx' )
            ];

when expecting...

    {1, a} x {1, a} -> {1, a, 1, a}
    {1, a, 1, a} x {1, a} -> {1, a, 1, a, 1, a}
    $VAR1 = [
              bless( [
                       1,
                       'a'
                     ], 'overx' ),
              bless( [
                       1,
                       'a',
                       1,
                       'a',
                       1,
                       'a'
                     ], 'overx' )
            ];

Other operators work as expected.

For instance, the following generates the expected result.

  perl -I. -Moverx -MData​::Dumper -e '$a = overx->new(1,'a'); print Dumper [$a, scalar(($a * $a) * $a)]'

While the following also fails​:

  perl -I. -Moverx -MData​::Dumper -e '$a = overx->new(1,'a'); print Dumper [$a, scalar(($a * $a) x $a)]'

I can reproduce the bug also in 5.8.8.

Perl Info

Flags:
    category=library
    severity=low
    module=overload

Site configuration information for perl 5.18.2:

Configured by Debian Project at Thu Mar 27 18:28:21 UTC 2014.

Summary of my perl5 (revision 5 version 18 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=3.2.0-58-generic, archname=x86_64-linux-gnu-thread-multi
    uname='linux brownie 3.2.0-58-generic #88-ubuntu smp tue dec 3 17:37:58 utc 2013 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.18 -Darchlib=/usr/lib/perl/5.18 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.18.2 -Dsitearch=/usr/local/lib/perl/5.18.2 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.18.2 -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.8.2', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=, so=so, useshrplib=true, libperl=libperl.so.5.18.2
    gnulibc_version='2.19'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - http://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - http://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - http://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: Set location of libperl.a to /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/prefix_changes - Fiddle with *PREFIX and variables written to the makefile
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/module_build_man_extensions - http://bugs.debian.org/479460 Adjust Module::Build manual page extensions for the Debian Perl policy
    DEBPKG:debian/prune_libs - http://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] http://bugs.debian.org/100195 Document the Net::SMTP 'Port' option
    DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/cpanplus_definstalldirs - http://bugs.debian.org/533707 Configure CPANPLUS to use the site directories by default.
    DEBPKG:debian/cpanplus_config_path - Save local versions of CPANPLUS::Config::System into /etc/perl.
    DEBPKG:debian/deprecate-with-apt - http://bugs.debian.org/702096 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
    DEBPKG:debian/patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.18.2-2ubuntu1 in patchlevel.h
    DEBPKG:debian/skip-kfreebsd-crash - http://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG:fixes/document_makemaker_ccflags - http://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - http://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/hurd_test_skip_stack - http://bugs.debian.org/650175 Disable failing GNU/Hurd tests dist/threads/t/stack.t
    DEBPKG:fixes/manpage_name_Test-Harness - http://bugs.debian.org/650451 [rt.cpan.org #73399] cpan/Test-Harness: add NAME headings in modules with POD
    DEBPKG:debian/makemaker-pasthru - http://bugs.debian.org/660195 [rt.cpan.org #28632] Make EU::MM pass LD through to recursive Makefile.PL invocations
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - http://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - http://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] http://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:fixes/net_ftp_failed_command - [rt.cpan.org #37700] http://bugs.debian.org/491062 Net::FTP: cope gracefully with a failed command
    DEBPKG:fixes/perlbug-patchlist - [3541c11] http://bugs.debian.org/710842 [perl #118433] Make perlbug look up the list of local patches at run time
    DEBPKG:fixes/module_metadata_security_doc - [68cdd4b] CVE-2013-1437 documentation fix
    DEBPKG:fixes/module_metadata_taint_fix - [bff978f] http://bugs.debian.org/722210 [rt.cpan.org #88576] untaint version, if needed, in Module::Metadata
    DEBPKG:fixes/IPC-SysV-spelling - http://bugs.debian.org/730558 [rt.cpan.org #86736] Fix spelling of IPC_CREAT in IPC-SysV documentation
    DEBPKG:fixes/fix-undef-source -


@INC for perl 5.18.2:
    /etc/perl
    /usr/local/lib/perl/5.18.2
    /usr/local/share/perl/5.18.2
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.18
    /usr/share/perl/5.18
    /usr/local/lib/site_perl
    /usr/local/lib/perl/5.18.1
    /usr/local/share/perl/5.18.1
    .


Environment for perl 5.18.2:
    HOME=/home/salva
    LANG=en_US.UTF-8
    LANGUAGE=en_US:en
    LC_ADDRESS=en_US.UTF-8
    LC_IDENTIFICATION=en_US.UTF-8
    LC_MEASUREMENT=en_US.UTF-8
    LC_MONETARY=en_US.UTF-8
    LC_NAME=en_US.UTF-8
    LC_NUMERIC=en_US.UTF-8
    LC_PAPER=en_US.UTF-8
    LC_TELEPHONE=en_US.UTF-8
    LC_TIME=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/scala/bin:/usr/local/scala/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From ambrus@math.bme.hu

On Thu May 08 07​:54​:20 2014, salva wrote​:

$ perl -I. -Moverx -MData​::Dumper -e '$a = overx->new(1,'a'); print Dumper [$a, scalar(($a x $a) x $a)]'

Could this be because the x operator in that case is parsed as list repetition, and only string repetition can be overloaded?

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @salva

----- Original Message -----

From​: Zsban Ambrus via RT <perlbug-followup@​perl.org>
To​: sfandino@​yahoo.com
Cc​:
Sent​: Saturday, May 10, 2014 3​:28 PM
Subject​: [perl #121827] overloading 'x' is broken

On Thu May 08 07​:54​:20 2014, salva wrote​:

$ perl -I. -Moverx -MData​::Dumper -e '$a =
overx->new(1,'a'); print Dumper [$a, scalar(($a x $a) x $a)]'

Could this be because the x operator in that case is parsed as list repetition,
and only string repetition can be overloaded?

No, the expression is wrapped by a "scalar" call.

In any case, the real issue is that the return value from the first
expression "escapes" from the array constructor. In other words, that is
like [$a, $b] returning ($a, [$b]). There is something here going really
wrong!

[p5pRT]

From @salva

The issue is caused by a 'pushmark' OP being prepended to the left-hand tree of the 'x' binary operator by the parser when the left expression is inside parenthesis.

When 'x' is not overloaded the code in pp_repeat undoes the pushmark and places the result on the parent frame, but when the operation is overloaded, the tryAMAGICbin_MG macro call just returns without removing the extra frame.

[p5pRT]

From [Unknown Contact. See original ticket]

The issue is caused by a 'pushmark' OP being prepended to the left-hand tree of the 'x' binary operator by the parser when the left expression is inside parenthesis.

When 'x' is not overloaded the code in pp_repeat undoes the pushmark and places the result on the parent frame, but when the operation is overloaded, the tryAMAGICbin_MG macro call just returns without removing the extra frame.

[p5pRT]

From @iabyn

On Sun, May 11, 2014 at 11​:30​:44AM +0200, Salvador Fandiño wrote​:

----- Original Message -----

From​: Zsban Ambrus via RT <perlbug-followup@​perl.org>
To​: sfandino@​yahoo.com
Cc​:
Sent​: Saturday, May 10, 2014 3​:28 PM
Subject​: [perl #121827] overloading 'x' is broken

On Thu May 08 07​:54​:20 2014, salva wrote​:

$ perl -I. -Moverx -MData​::Dumper -e '$a =
overx->new(1,'a'); print Dumper [$a, scalar(($a x $a) x $a)]'

Could this be because the x operator in that case is parsed as list repetition,
and only string repetition can be overloaded?

No, the expression is wrapped by a "scalar" call.

In any case, the real issue is that the return value from the first
expression "escapes" from the array constructor. In other words,
that is like [$a, $b] returning ($a, [$b]). There is something here
going really wrong!

In the 'x' case, an extra pushmark is happening after the '[ $a, ',
so that when anonlist is executed, it only collects the elements
to the right of '[ $a, ' into the anonymous list.

Whether the pushmark is extraneous, or whether its the fault of something
further along not popping the mark, I don't know.

Anyway, I haven't really got time to look any further into this yet, as it
isn't a 5.20 blocker.

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

[p5pRT]

From @cpansprout

On Mon May 12 04​:17​:13 2014, salva wrote​:

The issue is caused by a 'pushmark' OP being prepended to the left-
hand tree of the 'x' binary operator by the parser when the left
expression is inside parenthesis.

When 'x' is not overloaded the code in pp_repeat undoes the pushmark
and places the result on the parent frame, but when the operation is
overloaded, the tryAMAGICbin_MG macro call just returns without
removing the extra frame.

There were two stack bugs in pp_repeat, so I fixed them both in 3a100da. Then I added an optimisation in 82e4f30 that sidesteps the issue most of the time.

--

Father Chrysostomos

[p5pRT]

@cpansprout - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thanks for submitting this ticket

The issue should be resolved with the release today of Perl v5.22. If you find that the problem persists, feel free to reopen this ticket

--
Karl Williamson for the Perl 5 porters team

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'