t/op/lex.t: Failures in test #30 on multiple operating systems

[jkeenan]

We are starting to observe failures in t/op/lex.t in smoke-test reports generated on a variety of operating systems, including Linux and two different versions of OpenBSD.

A listing of these failing smoke-test reports is here.

Based on the git describe values in those reports, it seems that these failures began to occur after recent extensive revisions in toke.c were merged into blead. @arc

The test failures are intermittent but consistently occur in test #30 from that file. The relevant code in t/op/lext.t is:

fresh_perl_is(
    "00my sub\0",
    "Missing name in \"my sub\" at - line 1.\n",
    {},
    '[perl #129069] - "Missing name" warning and valgrind clean'
);

The RT cited in the description for that test is now found here: Fuzzer-detected use-after-free in Perl_yylex.

I am attaching two files which are excerpts from the logs/smokecurrent/log*.log files generated in two different smoke tests I have run on OpenBSD-6.4. However, when I do a regular make test_harness on blead on this platform, t/op/lex.t passes. So these failures are associated with the more stressful conditions present during smoke-testing. Note that while we have observed these failures on OpenBSD (@afresh1), we are also observing them on Linux.

Thank you very much.
Jim Keenan

ad1d31a5.openbsd.smoke.log.t-op-lex-t.failures.txt
bb6e2c77.openbsd.smoke.log.t-op-lex-t.failures.txt

$ ./perl -Ilib -V  
Summary of my perl5 (revision 5 version 31 subversion 6) configuration:
  Snapshot of: ad1d31a566e2cab4603d8d3f39b92fe418748ace
  Platform:
    osname=openbsd
    osvers=6.4
    archname=OpenBSD.amd64-openbsd-thread-multi
    uname='openbsd perl-reporter-04 6.4 generic.mp#364 amd64 '
    config_args='-des -Dusedevel -Duseithreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-pthread -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-pthread -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible OpenBSD Clang 6.0.0 (tags/RELEASE_600/final)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-pthread -Wl,-E  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/lib /usr/local/lib
    libs=-lpthread -lm -lutil -lc
    perllibs=-lpthread -lm -lutil -lc
    libc=/usr/lib/libc.so.92.5
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags='-DPIC -fPIC '
    lddlflags='-shared -fPIC  -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_DEVEL
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
  Locally applied patches:
    SMOKEad1d31a566e2cab4603d8d3f39b92fe418748ace
  Built under openbsd
  Compiled at Oct 25 2019 22:38:21
  %ENV:
    PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @INC:
    lib
    /usr/local/lib/perl5/site_perl/5.31.6/OpenBSD.amd64-openbsd-thread-multi
    /usr/local/lib/perl5/site_perl/5.31.6
    /usr/local/lib/perl5/5.31.6/OpenBSD.amd64-openbsd-thread-multi
    /usr/local/lib/perl5/5.31.6

[tonycoz]

Reproduced on Linux with valgrind:

tony@mars:.../perl/t$ PERL_RUNPERL_DEBUG='valgrind -q' ./perl -I../lib op/lex.t
1..36
ok 1
ok 2
...
ok 28 - [perl \#126482] Assert failure when mentioning a constant twice in a row
ok 29 - [perl \#129069] - no output and valgrind clean
not ok 30 - [perl \#129069] - "Missing name" warning and valgrind clean
# Failed test 30 - [perl \#129069] - "Missing name" warning and valgrind clean at ./test.pl line 1062
#      got "==24368== Invalid write of size 1\n==24368==    at 0x1EF414: yyl_sub (toke.c:5110)\n==24368==    by 0x208B0A: yyl_try (toke.c:8840)\n==24368==    by 0x20AF85: Perl_yylex (toke.c:9292)\n==24368==    by 0x220AF7: Perl_yyparse (perly.c:340)\n==24368==    by 0x1A117F: S_parse_body (perl.c:2527)\n==24368==    by 0x19F43A: perl_parse (perl.c:1818)\n==24368==    by 0x1554F9: main (perlmain.c:126)\n==24368==  Address 0x5f822f8 is 8 bytes inside a block of size 10 free\'d\n==24368==    at 0x4C2DDCF: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)\n==24368==    by 0x2CAE71: Perl_safesysrealloc (util.c:279)\n==24368==    by 0x34363F: Perl_sv_grow (sv.c:1596)\n==24368==    by 0x37E8A0: Perl_sv_gets (sv.c:8679)\n==24368==    by 0x1EB69E: S_filter_gets (toke.c:4678)\n==24368==    by 0x1D8B40: Perl_lex_next_chunk (toke.c:1353)\n==24368==    by 0x1D96E9: Perl_lex_read_space (toke.c:1588)\n==24368==    by 0x1DB4EA: Perl_skipspace_flags (toke.c:1900)\n==24368==    by 0x1EEF20: yyl_sub (toke.c:5076)\n==24368==    by 0x208B0A: yyl_try (toke.c:8840)\n==24368==    by 0x20AF85: Perl_yylex (toke.c:9292)\n==24368==    by 0x220AF7: Perl_yyparse (perly.c:340)\n==24368==  Block was alloc\'d at\n==24368==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)\n==24368==    by 0x2CAD1F: Perl_safesysmalloc (util.c:155)\n==24368==    by 0x343654: Perl_sv_grow (sv.c:1599)\n==24368==    by 0x3674BC: Perl_sv_setpvn (sv.c:4972)\n==24368==    by 0x384C00: Perl_newSVpvn (sv.c:9423)\n==24368==    by 0x1D5C81: Perl_lex_start (toke.c:773)\n==24368==    by 0x1A1121: S_parse_body (perl.c:2516)\n==24368==    by 0x19F43A: perl_parse (perl.c:1818)\n==24368==    by 0x1554F9: main (perlmain.c:126)\n==24368== \nMissing name in \"my sub\" at - line 1."
# expected "Missing name in \"my sub\" at - line 1."
# PROG: 
# 00my sub
# STATUS: 65280
ok 31 - [perl \#129336] - \#!perl -i argument handling
ok 32 - [perl \#128996] - use of PL_op after op is freed
...

If you create a file from the test case you can reproduce it outside the test script, making the result easier to read:

tony@mars:.../git/perl$ hexdump -C ../17223.pl
00000000  30 30 6d 79 20 73 75 62  00                       |00my sub.|
00000009
tony@mars:.../git/perl$ valgrind -q ./perl ../17223.pl 
==24496== Invalid write of size 1
==24496==    at 0x1EF414: yyl_sub (toke.c:5110)
==24496==    by 0x208B0A: yyl_try (toke.c:8840)
==24496==    by 0x20AF85: Perl_yylex (toke.c:9292)
==24496==    by 0x220AF7: Perl_yyparse (perly.c:340)
==24496==    by 0x1A117F: S_parse_body (perl.c:2527)
==24496==    by 0x19F43A: perl_parse (perl.c:1818)
==24496==    by 0x1554F9: main (perlmain.c:126)
==24496==  Address 0x5f817d8 is 8 bytes inside a block of size 10 free'd
==24496==    at 0x4C2DDCF: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24496==    by 0x2CAE71: Perl_safesysrealloc (util.c:279)
==24496==    by 0x34363F: Perl_sv_grow (sv.c:1596)
==24496==    by 0x37E8A0: Perl_sv_gets (sv.c:8679)
==24496==    by 0x1EB69E: S_filter_gets (toke.c:4678)
==24496==    by 0x1D8B40: Perl_lex_next_chunk (toke.c:1353)
==24496==    by 0x1D96E9: Perl_lex_read_space (toke.c:1588)
==24496==    by 0x1DB4EA: Perl_skipspace_flags (toke.c:1900)
==24496==    by 0x1EEF20: yyl_sub (toke.c:5076)
==24496==    by 0x208B0A: yyl_try (toke.c:8840)
==24496==    by 0x20AF85: Perl_yylex (toke.c:9292)
==24496==    by 0x220AF7: Perl_yyparse (perly.c:340)
==24496==  Block was alloc'd at
==24496==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24496==    by 0x2CAD1F: Perl_safesysmalloc (util.c:155)
==24496==    by 0x343654: Perl_sv_grow (sv.c:1599)
==24496==    by 0x3674BC: Perl_sv_setpvn (sv.c:4972)
==24496==    by 0x384C00: Perl_newSVpvn (sv.c:9423)
==24496==    by 0x1D5C81: Perl_lex_start (toke.c:773)
==24496==    by 0x1A1121: S_parse_body (perl.c:2516)
==24496==    by 0x19F43A: perl_parse (perl.c:1818)
==24496==    by 0x1554F9: main (perlmain.c:126)
==24496== 
Missing name in "my sub" at ../17223.pl line 1.

[tonycoz]

Fixed by ffa4682, I don't see any other similar reversions in the rest of the refactor.

[jkeenan]

This smoke test report on OpenBSD-6.6 confirms that we've gotten past the failure in t/op/lex.t: http://perl5.test-smoke.org/report/98330