grep and map lead to Perl_op_free: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.

[p5pRT]

Migrated from rt.perl.org#126258 (status was 'resolved')

Searchable as RT126258$

[p5pRT]

From @dcollinsn

Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in the perl interpreter, but which (correctly) errors out in non-debugging perl. The simplest testcase is the 9-character file​:

grep$0,0}

In a not-debugging perl, the output is​:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.

In a debugging perl, the output is​:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl​: op.c​:717​: Perl_op_free​: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.
Aborted

Other failing testcases that might help you understand what's going on here​:
grep$0,0}
map\0,0} <<--- Literal backslash
map\n0,0} <<--- Literal newline
0%grep{.}0
0%N{.}0%grep%0,0
map\nmap,0 <<--- Literal newline

**GDB**
dcollins@​nightshade64​:~/perldebug$ gdb --args ./perl -Ilib ../uncat/f3i000003 GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+​: GNU GPL version 3 or later <http​://gnu.org/licenses/gpl.html>
This is free software​: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see​:
<http​://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at​:
<http​://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perl...done.
(gdb) run
Starting program​: /home/dcollins/perldebug/perl -Ilib ../uncat/f3i000003
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl​: op.c​:717​: Perl_op_free​: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf4107 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cf54e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6ced226 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6ced2d2 in __assert_fail ()
  from /lib/x86_64-linux-gnu/libc.so.6
#4 0x000000000043f359 in Perl_op_free (o=0x12118b0) at op.c​:717
#5 0x0000000000529301 in perl_destruct (my_perl=<optimized out>)
  at perl.c​:777
#6 0x000000000042c660 in main (argc=3, argv=0x7fffffffe648,
  env=0x7fffffffe668) at perlmain.c​:127
(gdb) frame 4
#4 0x000000000043f359 in Perl_op_free (o=0x12118b0) at op.c​:717
717 assert(!(o->op_private & ~PL_op_private_valid[type]));
(gdb) info locals
type = 169
defer_ix = <optimized out>
defer_stack_alloc = <optimized out>
defer_stack = <optimized out>
__PRETTY_FUNCTION__ = "Perl_op_free"

**PERL -V**
dcollins@​nightshade64​:~/perldebug$ ./perl -V
Summary of my perl5 (revision 5 version 23 subversion 4) configuration​:
  Commit id​: 94757bf
  Platform​:
  osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux-ld
  uname='linux nightshade64 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 gnulinux '
  config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='ccache afl-gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='4.9.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='ccache afl-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
  USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
  USE_PERLIO USE_PERL_ATOF
  Built under linux
  Compiled at Oct 2 2015 22​:41​:42
  @​INC​:
  /usr/local/perl-afl/lib/site_perl/5.23.4/x86_64-linux-ld
  /usr/local/perl-afl/lib/site_perl/5.23.4
  /usr/local/perl-afl/lib/5.23.4/x86_64-linux-ld
  /usr/local/perl-afl/lib/5.23.4
  .

[p5pRT]

From @iabyn

On Sun, Oct 04, 2015 at 04​:13​:28AM -0700, Dan Collins wrote​:

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in the perl interpreter, but which (correctly) errors out in non-debugging perl. The simplest testcase is the 9-character file​:

grep$0,0}
[snip]
In a debugging perl, the output is​:
Unmatched right curly bracket at ../uncat/f3i000003 line 1, at end of line
syntax error at ../uncat/f3i000003 line 1, near "0}"
Execution of ../uncat/f3i000003 aborted due to compilation errors.
perl​: op.c​:717​: Perl_op_free​: Assertion `!(o->op_private & ~PL_op_private_valid[type])' failed.
Aborted

I've pushed a general fix for these sorts of issues with​:

commit 09681a1
Author​: David Mitchell <davem@​iabyn.com>
AuthorDate​: Thu Nov 19 15​:49​:03 2015 +0000
Commit​: David Mitchell <davem@​iabyn.com>
CommitDate​: Thu Nov 19 16​:00​:31 2015 +0000

  op_free()​: don't assert op_private ok when erred
 
  [perl #126258]
 
  op_free includes an assert to make sure each op's op_private field
  only has the flags set that are expected for that op. It's a thing I added
  at the same time I added the regen/op_private mechanism, and is more a
  general "make sure people are only setting the flags we know about" test.
 
  However, if the op tree is being freed after a compilation error, some
  op's flags may be in an inconsistent state; so skip the assert in that case.
 
  e.g.
 
  perl -e 'grep$0,0}'

--
Modern art​:
  "That's easy, I could have done that!"
  "Ah, but you didn't!"

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @mauke

Am 19.11.2015 um 17​:02 schrieb Dave Mitchell​:

I've pushed a general fix for these sorts of issues with​:

commit 09681a1
Author​: David Mitchell <davem@​iabyn.com>
AuthorDate​: Thu Nov 19 15​:49​:03 2015 +0000
Commit​: David Mitchell <davem@​iabyn.com>
CommitDate​: Thu Nov 19 16​:00​:31 2015 +0000

 op\_free\(\)​: don't assert op\_private ok when erred

 \[perl \#126258\]

 op\_free includes an assert to make sure each op's op\_private field
 only has the flags set that are expected for that op\. It's a thing I added
 at the same time I added the regen/op\_private mechanism\, and is more a
 general "make sure people are only setting the flags we know about" test\.

 However\, if the op tree is being freed after a compilation error\, some
 op's flags may be in an inconsistent state; so skip the assert in that case\.

 e\.g\.

     perl \-e 'grep$0\,0\}'

From this description I would have expected something like​:

  if (!(PL_parser && PL_parser->error_count)) {
  assert(...);
  }

i.e. assert unless we're currently parsing and there has been an error.

But the patch actually does

  if (PL_parser && !PL_parser->error_count) {
  assert(...);
  }

Is this intentional?

--
Lukas Mai <plokinom@​gmail.com>

[p5pRT]

From @iabyn

On Thu, Nov 19, 2015 at 09​:22​:03PM +0100, Lukas Mai wrote​:

From this description I would have expected something like​:

if \(\!\(PL\_parser && PL\_parser\->error\_count\)\) \{
    assert\(\.\.\.\);
\}

i.e. assert unless we're currently parsing and there has been an error.

But the patch actually does

if \(PL\_parser && \!PL\_parser\->error\_count\) \{
    assert\(\.\.\.\);
\}

Is this intentional?

Yes. If there's a compilation error in the main body of the program
(as opposed to code that was evalled/required/used), then by the time the
defective optree is is freed, PL_parser is already NULL​: so we have
no way of knowing whether it compiled ok or not; so always skip on a NULL
PL_parser.

--
"I do not resent criticism, even when, for the sake of emphasis,
it parts for the time with reality".
  -- Winston Churchill, House of Commons, 22nd Jan 1941.

[p5pRT]

@iabyn - Status changed from 'open' to 'resolved'