Assert fail in S_sublex_done without other symptoms

[p5pRT]

Migrated from rt.perl.org#128255 (status was 'resolved')

Searchable as RT128255$

[p5pRT]

From @dcollinsn

Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusequadmath -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in debug buids of the perl interpreter. The testcase is the file below. On normal builds, this runs normally (albeit with an expected warning). On debug builds, this returns an assert fail.

dcollins@​nightshade64​:~/perl$ ./perl -Ilib /usr/local/perl-afl/out/allcrash/f1i000029
Use of comma-less variable list is deprecated at /usr/local/perl-afl/out/allcrash/f1i000029 line 3.
Final $ should be \$ or $name at /usr/local/perl-afl/out/allcrash/f1i000029 line 3, within string
syntax error at /usr/local/perl-afl/out/allcrash/f1i000029 line 3, next char /
Format not terminated at /usr/local/perl-afl/out/allcrash/f1i000029 line 4, at end of line
syntax error at /usr/local/perl-afl/out/allcrash/f1i000029 line 4, at EOF
Execution of /usr/local/perl-afl/out/allcrash/f1i000029 aborted due to compilation errors.

dcollins@​nightshade64​:~/perl$ cd ../perldebug/
dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib /usr/local/perl-afl/out/allcrash/f1i000029
Use of comma-less variable list is deprecated at /usr/local/perl-afl/out/allcrash/f1i000029 line 3.
perl​: toke.c​:2450​: S_sublex_done​: Assertion `(PL_parser->lex_inwhat) == OP_SUBST || (PL_parser->lex_inwhat) == OP_TRANS' failed.
Aborted

dcollins@​nightshade64​:~/perldebug$ cat /usr/local/perl-afl/out/allcrash/f1i000029
format=
@​
0"$"dcollins@​nightshade64​:~/perldebug$ od -c /usr/local/perl-afl/out/allcrash/f1i000029
0000000 f o r m a t = \n @​ \n 0 " $ "
0000016

Debugging tool output is below. A git bisect was performed and reported the following.

5aa9185 is the first bad commit
commit 5aa9185
Author​: Father Chrysostomos <sprout@​cpan.org>
Date​: Mon Dec 30 21​:18​:54 2013 -0800

  toke.c​: Remove redundant check

  (and turn it into an assertion.)

  PL_lex_repl can only be set here if we are parsing a substitution
  (OP_SUBST) or transliteration (OP_TRANS). I believe this check
  has been unnecessary since perl 5.000, which it was added.

:100644 100644 ca306f1455ba8f2c02acd3d7eff3e5490706de84 8ac0f31fb4f85c4ab453b52ec75f92900c448292 M toke.c
bisect run success

**GDB**

dcollins@​nightshade64​:~/perldebug$ gdb --args ./miniperl -Ilib /usr/local/perl-afl/out/allcrash/f1i000029
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+​: GNU GPL version 3 or later <http​://gnu.org/licenses/gpl.html>
This is free software​: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see​:
<http​://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at​:
<http​://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./miniperl...done.
(gdb) run
Starting program​: /home/dcollins/perldebug/miniperl -Ilib /usr/local/perl-afl/out/allcrash/f1i000029
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Use of comma-less variable list is deprecated at /usr/local/perl-afl/out/allcrash/f1i000029 line 3.
miniperl​: toke.c​:2450​: S_sublex_done​: Assertion `(PL_parser->lex_inwhat) == OP_SUBST || (PL_parser->lex_inwhat) == OP_TRANS' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cfa8fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6cf23a7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6cf2452 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x000000000047ea40 in S_sublex_done () at toke.c​:2450
#5 0x000000000048b6d5 in Perl_yylex () at toke.c​:4728
#6 0x00000000004ba030 in Perl_yyparse (gramtype=258) at perly.c​:332
#7 0x00000000004475b4 in S_parse_body (env=0x0, xsinit=0x727694 <xs_init>) at perl.c​:2365
#8 0x0000000000446052 in perl_parse (my_perl=0xa7f010, xsinit=0x727694 <xs_init>, argc=3,
  argv=0x7fffffffe5f8, env=0x0) at perl.c​:1681
#9 0x00000000007275f3 in main (argc=3, argv=0x7fffffffe5f8, env=0x7fffffffe618)
  at miniperlmain.c​:120
(gdb) f 4
#4 0x000000000047ea40 in S_sublex_done () at toke.c​:2450
2450 assert (PL_lex_inwhat == OP_SUBST || PL_lex_inwhat == OP_TRANS);
(gdb) l
2445 }
2446
2447 /* Is there a right-hand side to take care of? (s//RHS/ or tr//RHS/) */
2448 assert(PL_lex_inwhat != OP_TRANSR);
2449 if (PL_lex_repl) {
2450 assert (PL_lex_inwhat == OP_SUBST || PL_lex_inwhat == OP_TRANS);
2451 PL_linestr = PL_lex_repl;
2452 PL_lex_inpat = 0;
2453 PL_bufend = PL_bufptr = PL_oldbufptr = PL_oldoldbufptr = PL_linestart = SvPVX(PL_linestr);
2454 PL_bufend += SvCUR(PL_linestr);

**VALGRIND**

No reported memory management errors.

**PERL -V**

dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 25 subversion 2) configuration​:
  Commit id​: c29dfc6
  Platform​:
  osname=linux, osvers=4.5.0-2-amd64, archname=x86_64-linux-ld
  uname='linux nightshade64 4.5.0-2-amd64 #1 smp debian 4.5.3-2 (2016-05-08) x86_64 gnulinux '
  config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache gcc-6.1 -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='ccache gcc-6.1', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='6.1.0', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='ccache gcc-6.1', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/local/lib/gcc/x86_64-pc-linux-gnu/6.1.0/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.22.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.22'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
  USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES
  USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
  USE_PERLIO USE_PERL_ATOF
  Built under linux
  Compiled at May 26 2016 17​:57​:37
  @​INC​:
  lib
  /usr/local/perl-afl/lib/site_perl/5.25.2/x86_64-linux-ld
  /usr/local/perl-afl/lib/site_perl/5.25.2
  /usr/local/perl-afl/lib/5.25.2/x86_64-linux-ld
  /usr/local/perl-afl/lib/5.25.2
  /usr/local/perl-afl/lib/site_perl/5.25.1
  /usr/local/perl-afl/lib/site_perl/5.24.0
  /usr/local/perl-afl/lib/site_perl
  .

[p5pRT]

From @iabyn

On Thu, May 26, 2016 at 05​:45​:05PM -0700, Dan Collins wrote​:

dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib /usr/local/perl-afl/out/allcrash/f1i000029
Use of comma-less variable list is deprecated at /usr/local/perl-afl/out/allcrash/f1i000029 line 3.
perl​: toke.c​:2450​: S_sublex_done​: Assertion `(PL_parser->lex_inwhat) == OP_SUBST || (PL_parser->lex_inwhat) == OP_TRANS' failed.
Aborted

dcollins@​nightshade64​:~/perldebug$ cat /usr/local/perl-afl/out/allcrash/f1i000029
format=
@​

Fixed by the below​:

  commit 59a08c7
  Author​: David Mitchell <davem@​iabyn.com>
  AuthorDate​: Fri Jul 8 09​:48​:04 2016 +0100
  Commit​: David Mitchell <davem@​iabyn.com>
  CommitDate​: Fri Jul 8 09​:48​:04 2016 +0100

  RT #128255​: Assert fail in S_sublex_done
 
  Some code that handles deprecated behaviour in formats was triggering
  an assertion. This​:
 
  format STDOUT =
  @​
  0"$x"
 
  gave this warning​:
 
  Use of comma-less variable list is deprecated
 
  but then gave this panic​:
 
  toke.c​:2457​: S_sublex_done​: Assertion `(PL_parser->lex_inwhat) ==
  OP_SUBST || (PL_parser->lex_inwhat) == OP_TRANS' failed.
 
  This is due to the lexer calling scan_str(), then backing off and doing
  the warning and returning a comma, then on the next token get, calling
  scan_str() again. Because scan_str() has been called twice, the
  second time it extracts the string to PL_sublex_info.repl rather than
  PL_lex_stuff, as it does with things like s/foo/bar/ and tr/abc/ABC/.
  Later an assert that PL_sublex_info.repl is only set for a s/// or tr///
  fails.
 
  The solution seems to be to check and return a comma *before*
  trying to call scan_str().

--
I before E. Except when it isn't.

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

@iabyn - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'