grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed. #14963

p5pRT · 2015-10-04T11:13:27Z

Migrated from rt.perl.org#126258 (status was 'resolved')

Searchable as RT126258$

p5pRT · 2015-10-04T11:13:27Z

From @dcollinsn

Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in the perl interpreter, but which (correctly) errors out in non-debugging perl. The simplest testcase is the 9-character file:

grep$0,0}

In a not-debugging perl, the output is:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.

In a debugging perl, the output is:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl: op.c:717: Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.
Aborted

Other failing testcases that might help you understand what's going on here:
grep$0,0}
map\0,0} <<--- Literal backslash
map\n0,0} <<--- Literal newline
0%grep{.}0
0%N{.}0%grep%0,0
map\nmap,0 <<--- Literal newline

**GDB**
dcollins@nightshade64:~/perldebug$ gdb --args ./perl -Ilib ../uncat/f3i000003 GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perl...done.
(gdb) run
Starting program: /home/dcollins/perldebug/perl -Ilib ../uncat/f3i000003
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl: op.c:717: Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cf54e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6ced226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6ced2d2 in __assert_fail ()
from /lib/x86_64-linux-gnu/libc.so.6
#4 0x000000000043f359 in Perl_op_free (o=0x12118b0) at op.c:717
#5 0x0000000000529301 in perl_destruct (my_perl=<optimized out>)
at perl.c:777
#6 0x000000000042c660 in main (argc=3, argv=0x7fffffffe648,
env=0x7fffffffe668) at perlmain.c:127
(gdb) frame 4
#4 0x000000000043f359 in Perl_op_free (o=0x12118b0) at op.c:717
717 assert(!(o->op_private & ~PL_op_private_valid[type]));
(gdb) info locals
type = 169
defer_ix = <optimized out>
defer_stack_alloc = <optimized out>
defer_stack = <optimized out>
__PRETTY_FUNCTION__ = "Perl_op_free"

**PERL -V**
dcollins@nightshade64:~/perldebug$ ./perl -V
Summary of my perl5 (revision 5 version 23 subversion 4) configuration:
Commit id: 94757bf
Platform:
osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux-ld
uname='linux nightshade64 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 gnulinux '
config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
use64bitint=define, use64bitall=define, uselongdouble=define
usemymalloc=n, bincompat5005=undef
Compiler:
cc='ccache afl-gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-g',
cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
alignbytes=16, prototype=define
Linker and Libraries:
ld='ccache afl-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.19'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl):
Compile-time options: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
USE_PERLIO USE_PERL_ATOF
Built under linux
Compiled at Oct 2 2015 22:41:42
@INC:
/usr/local/perl-afl/lib/site_perl/5.23.4/x86_64-linux-ld
/usr/local/perl-afl/lib/site_perl/5.23.4
/usr/local/perl-afl/lib/5.23.4/x86_64-linux-ld
/usr/local/perl-afl/lib/5.23.4
.

p5pRT · 2015-11-19T16:03:00Z

From @iabyn

On Sun, Oct 04, 2015 at 04:13:28AM -0700, Dan Collins wrote:

I have compiled bleadperl with the afl-gcc compiler using:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in the perl interpreter, but which (correctly) errors out in non-debugging perl. The simplest testcase is the 9-character file:

grep$0,0}
[snip]
In a debugging perl, the output is:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl: op.c:717: Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.
Aborted

I've pushed a general fix for these sorts of issues with:

commit 09681a1
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Thu Nov 19 15:49:03 2015 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Thu Nov 19 16:00:31 2015 +0000

op_free(): don't assert op_private ok when erred

[perl #126258]

op_free includes an assert to make sure each op's op_private field
only has the flags set that are expected for that op. It's a thing I added
at the same time I added the regen/op_private mechanism, and is more a
general "make sure people are only setting the flags we know about" test.

However, if the op tree is being freed after a compilation error, some
op's flags may be in an inconsistent state; so skip the assert in that case.

e.g.

perl -e 'grep$0,0}'

--
Modern art:
"That's easy, I could have done that!"
"Ah, but you didn't!"

p5pRT · 2015-11-19T16:03:01Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2015-11-19T20:22:35Z

From @mauke

Am 19.11.2015 um 17:02 schrieb Dave Mitchell:

I've pushed a general fix for these sorts of issues with:

commit 09681a1
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Thu Nov 19 15:49:03 2015 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Thu Nov 19 16:00:31 2015 +0000
 op\_free&#8203;: don't assert op\_private ok when erred

 \[perl \#126258\]

 op\_free includes an assert to make sure each op's op\_private field
 only has the flags set that are expected for that op\. It's a thing I added
 at the same time I added the regen/op\_private mechanism\, and is more a
 general "make sure people are only setting the flags we know about" test\.

 However\, if the op tree is being freed after a compilation error\, some
 op's flags may be in an inconsistent state; so skip the assert in that case\.

 e\.g\.

     perl \-e 'grep$0\,0\}'

From this description I would have expected something like:

if (!(PL_parser && PL_parser->error_count)) {
assert(...);
}

i.e. assert unless we're currently parsing and there has been an error.

But the patch actually does

if (PL_parser && !PL_parser->error_count) {
assert(...);
}

Is this intentional?

--
Lukas Mai <plokinom@gmail.com>

p5pRT · 2015-11-19T23:27:35Z

From @iabyn

On Thu, Nov 19, 2015 at 09:22:03PM +0100, Lukas Mai wrote:

From this description I would have expected something like:
if $\!\(PL\_parser && PL\_parser\->error\_count$\) \{
    assert$\.\.\.$;
\}
i.e. assert unless we're currently parsing and there has been an error.

But the patch actually does
if $PL\_parser && \!PL\_parser\->error\_count$ \{
    assert$\.\.\.$;
\}
Is this intentional?

Yes. If there's a compilation error in the main body of the program
(as opposed to code that was evalled/required/used), then by the time the
defective optree is is freed, PL_parser is already NULL: so we have
no way of knowing whether it compiled ok or not; so always skip on a NULL
PL_parser.

--
"I do not resent criticism, even when, for the sake of emphasis,
it parts for the time with reality".
-- Winston Churchill, House of Commons, 22nd Jan 1941.

p5pRT · 2015-11-23T11:30:39Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT closed this Nov 23, 2015

p5pRT added the Severity Low label Oct 19, 2019

Perl / perl5 Public

grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed. #14963

grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed. #14963

p5pRT commented Oct 4, 2015

p5pRT commented Oct 4, 2015

p5pRT commented Nov 19, 2015

p5pRT commented Nov 19, 2015

p5pRT commented Nov 19, 2015

p5pRT commented Nov 19, 2015

p5pRT commented Nov 23, 2015

Perl / perl5 Public

grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed. #14963

grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed. #14963

Comments

p5pRT commented Oct 4, 2015

p5pRT commented Oct 4, 2015

From @dcollinsn

p5pRT commented Nov 19, 2015

From @iabyn

p5pRT commented Nov 19, 2015

p5pRT commented Nov 19, 2015

From @mauke

p5pRT commented Nov 19, 2015

From @iabyn

p5pRT commented Nov 23, 2015