Assert fail in S_find_uninit_var (via Perl_pp_multideref) without other symptoms: $ISA[0][0]

[p5pRT]

Migrated from rt.perl.org#128253 (status was 'resolved')

Searchable as RT128253$

[p5pRT]

From @dcollinsn

Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusequadmath -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in debug buids of the perl interpreter. The testcase is the file below. On normal builds, this runs normally (albeit with an expected warning). On debug builds, this returns an assert fail.

dcollins@​nightshade64​:~/perl$ ./perl -Ilib -W -e '$ISA[0][0]'
Useless use of array element in void context at -e line 1.
Use of uninitialized value in array element at -e line 1.
****100 additional copies of the preceding line trimmed****
Recursive inheritance detected in package 'main' at -e line 1.

dcollins@​nightshade64​:~/perl$ cd ../perldebug/
dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib -e '$ISA[0][0]'
Recursive inheritance detected in package 'main' at -e line 1.

dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib -W -e '$ISA[0][0]'
Useless use of array element in void context at -e line 1.
perl​: sv.c​:16093​: S_find_uninit_var​: Assertion `is_hv' failed.
Aborted

Debugging tool output is below. A git bisect was performed and reported the following.

fedf30e is the first bad commit
commit fedf30e
Author​: David Mitchell <davem@​iabyn.com>
Date​: Fri Oct 24 16​:26​:38 2014 +0100

  Add OP_MULTIDEREF

  This op is an optimisation for any series of one or more array or hash
  lookups and dereferences, where the key/index is a simple constant or
  package/lexical variable. If the first-level lookup is of a simple
  array/hash variable or scalar ref, then that is included in the op too.

  So all of the following are replaced with a single op​:

  $h{foo}
  $a[$i]
  $a[5][$k][$i]
  $r->{$k}
  local $a[0][$i]
  exists $a[$i]{$k}
  delete $h{foo}

  while these aren't​:

  $a[0] already handled by OP_AELEMFAST
  $a[$x+1] not a simple index

  and these are partially replaced​:

  (expr)->[0]{$k} the bit following (expr) is replaced
  $h{foo}[$x+1][0] the first and third lookups are each done with
  a multideref op, while the $x+1 expression and
  middle lookup are done by existing add, aelem etc
  ops.

  Up until now, aggregate dereferencing has been very heavyweight in ops; for
  example, $r->[0]{$x} is compiled as​:

  gv[*r] s
  rv2sv sKM/DREFAV,1
  rv2av[t2] sKR/1
  const[IV 0] s
  aelem sKM/DREFHV,2
  rv2hv sKR/1
  gvsv[*x] s
  helem vK/2

  When executing this, in addition to the actual calls to av_fetch() and
  hv_fetch(), there is a lot of overhead of pushing SVs on and off the
  stack, and calling lots of little pp() functions from the runops loop
  (each with its potential indirect branch miss).

  The multideref op avoids that by running all the code in a loop in a
  switch statement. It makes use of the new UNOP_AUX type to hold an array
  of

  typedef union {
  PADOFFSET pad_offset;
  SV *sv;
  IV iv;
  UV uv;
  } UNOP_AUX_item;

  In something like $a[7][$i]{foo}, the GVs or pad offsets for @​a and $i are
  stored as items in the array, along with a pointer to a const SV holding
  'foo', and the UV 7 is stored directly. Along with this, some UVs are used
  to store a sequence of actions (several actions are squeezed into a single
  UV).

  Then the main body of pp_multideref is a big while loop round a switch,
  which reads actions and values from the AUX array. The two big branches in
  the switch are ones that are affectively unrolled (/DREFAV, rv2av, aelem)
  and (/DREFHV, rv2hv, helem) triplets. The other branches are various entry
  points that handle retrieving the different types of initial value; for
  example 'my %h; $h{foo}' needs to get %h from the pad, while '(expr)->{foo}'
  needs to pop expr off the stack.

  Note that there is a slight complication with /DEREF; in the example above
  of $r->[0]{$x}, the aelem op is actually

  aelem sKM/DREFHV,2

  which means that the aelem, after having retrieved a (possibly undef)
  value from the array, is responsible for autovivifying it into a hash,
  ready for the next op. Similarly, the rv2sv that retrieves $r from the
  typeglob is responsible for autovivifying it into an AV. This action
  of doing the next op's work for it complicates matters somewhat. Within
  pp_multideref, the autovivification action is instead included as the
  first step of the current action.

  In terms of benchmarking with Porting/bench.pl, a simple lexical
  $a[$i][$j] shows a reduction of approx 40% in numbers of instructions
  executed, while $r->[0][0][0] uses 54% fewer. The speed-up for hash
  accesses is relatively more modest, since the actual hash lookup (i.e.
  hv_fetch()) is more expensive than an array lookup. A lexical $h{foo}
  uses 10% fewer, while $r->{foo}{bar}{baz} uses 34% fewer instructions.

  Overall,
  bench.pl --tests='/expr​::(array|hash)/' ...
  gives​:
  PRE POST
  ------ ------

  Ir 100.00 145.00
  Dr 100.00 165.30
  Dw 100.00 175.74
  COND 100.00 132.02
  IND 100.00 171.11

  COND_m 100.00 127.65
  IND_m 100.00 203.90

  with cache misses unchanged at 100%.

  In general, the more lookups done, the bigger the proportionate saving.

:100644 100644 4775d8ee963455f92943ec5801667f52c27d4cc8 7d04e1f4a95ce2052bf6d2edf5bb37f626fa81bb M MANIFEST
:040000 040000 ead841309200671a901d9eba96adcd7b99dc8c18 6d89fbbb6a0630992a64357e75a117adc80db012 M Porting
:040000 040000 e381210f04b7f4f650269461bd13281cdf504208 689f04448c02694a9edd98b34c9c21afee60a434 M dist
:100644 100644 daeedf493f0295b47fdd157659e856e0f62fd9c1 9abfbb12bc89dfa97d544956b9c65846a3e79b4c M dump.c
:100644 100644 26d893d13d9147f791127676fa7970f8bdb04d34 c7b283b9f663ccf27f1b8608861079acd1170c50 M embed.fnc
:100644 100644 7108b3e331a35a9441f81cefb18988cf8a1c50ce 02d25be1cbaf1a81d95e97f8f6a59d4a3f280021 M embed.h
:100644 100644 9e4a910a57e54c8e8caa5c46b2e6d712e1d397f3 32a8b9b32710211bb54be950d275df4a707d649d M embedvar.h
:040000 040000 78f77af5676708a3d6d481c1984bebb46dd4cf40 e2e7280b7c1e47936223dbb0e100f54ff9bcff2c M ext
:100644 100644 39eac06454ad06aee3598cc935d2c3553d84fc12 ffb1172c9f1c7eac9659d1421b77749e995874b8 M intrpvar.h
:040000 040000 a18a0cc712b77b7f80c32476b1d6b2e6df1f71d3 e95253c1697d54f421f58ba92a7885f4ffc6bddb M lib
:100644 100644 f34e9326f7bfe0f99ef0f3d83528b1778b07baed 16ebd4244b24ffcd677999b219be3160b9d8f9ce M op.c
:100644 100644 61a382fb7614ac6f86320af4afb6785aa9963c55 9e60beb64dbd900c59eebac2fbaf6f45c6ab1728 M op.h
:100644 100644 e67318ffc82e60e765e8a16e5abb59037dd4119f 4266c49cf6d1c548842aebdf7e8c0f36cb084043 M opcode.h
:100644 100644 dce44f19e98b9ca7953da952300a810a26c25df1 1d259a15ddfb4a640d1e0c1cea5dfad469d65a01 M opnames.h
:100644 100644 2a77522c5efda6db3bb0b9413123c556193d61c5 ac674c1aa95c304715611c37b7f1f3f8f5923090 M perl.h
:100644 100644 97ad595119fc580b261a29e972b510ebf1f2efbb 67729994634c4e8d08c955f7c4b6797012f44451 M pp.c
:100644 100644 35493eb1b48215c53225ef53a21079410186941f 24bf8e9086954fb901ace8b99ce1854fb9d8fbfb M pp_hot.c
:100644 100644 6959357dd25d306192727b5a9625736aaf8c8e26 074f4ab8a36df0d182883178737a1b3f8c4040a1 M pp_proto.h
:100644 100644 eb2ba5a9adf0f5fe2853dd1fa7ee967eb6d4c30b f2be12dff5e07838dd3f840580834e041ab41c02 M proto.h
:040000 040000 bc56c3274ae52f12fa3efe24b8a5b353fdb4d63b 86365afc8cb53ffbfecb4c4c597d9d830b0caedd M regen
:100644 100644 5a73d9565ce71cf33bd3b01e1dcb486927c22d33 b08899fc9992a106cab017c02e21a1bc3c6abdb8 M sv.c
:040000 040000 4243a5a21c1c1f030a71cd0988c01286e5e99b14 7b899063862060c43f47a069fa094dd30517fdfe M t
bisect run success

**GDB**

dcollins@​nightshade64​:~/perldebug$ gdb --args ./perl -Ilib -W -e '(($ISA[0][0]))'
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+​: GNU GPL version 3 or later <http​://gnu.org/licenses/gpl.html>
This is free software​: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see​:
<http​://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at​:
<http​://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perl...done.
(gdb) run
Starting program​: /home/dcollins/perldebug/perl -Ilib -W -e \(\(\$ISA\[0\]\[0\]\)\)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Useless use of array element in void context at -e line 1.
perl​: sv.c​:16093​: S_find_uninit_var​: Assertion `is_hv' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cfa8fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6cf23a7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6cf2452 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000618df3 in S_find_uninit_var (obase=0xabc100, uninit_sv=0xa9e048, match=false,
  desc_p=0x7fffffffd518) at sv.c​:16093
#5 0x0000000000619c9d in Perl_report_uninit (uninit_sv=0xa9e048) at sv.c​:16392
#6 0x00000000005d2003 in Perl_sv_2pv_flags (sv=0xa9e048, lp=0x7fffffffd890, flags=34)
  at sv.c​:3179
#7 0x00000000005945b2 in Perl_hv_common (hv=0xa9dc88, keysv=0xa9e048, key=0x0, klen=0,
  flags=0, action=0, val=0x0, hash=0) at hv.c​:385
#8 0x000000000047aaa4 in S_gv_stashsvpvn_cached (namesv=0xa9e048, name=0x0, namelen=0,
  flags=0) at gv.c​:1453
#9 0x000000000047ae23 in Perl_gv_stashsv (sv=0xa9e048, flags=0) at gv.c​:1505
#10 0x0000000000581f7b in S_mro_get_linear_isa_dfs (stash=0xa9dec8, level=0) at mro_core.c​:280
#11 0x000000000058269e in Perl_mro_get_linear_isa (stash=0xa9dec8) at mro_core.c​:413
#12 0x000000000058428f in Perl_mro_isa_changed_in (stash=0xa9dec8) at mro_core.c​:652
#13 0x0000000000575649 in Perl_magic_clearisa (sv=0x0, mg=0xabbc90) at mg.c​:1731
#14 0x0000000000575104 in Perl_magic_setisa (sv=0xab2728, mg=0xabbc90) at mg.c​:1691
#15 0x000000000056dfa1 in Perl_mg_set (sv=0xab2728) at mg.c​:277
#16 0x00000000005a2715 in Perl_av_store (av=0xab2728, key=0, val=0xa9e048) at av.c​:381
#17 0x00000000005a2216 in Perl_av_fetch (av=0xab2728, key=0, lval=1) at av.c​:279
#18 0x00000000005b5c6d in Perl_pp_multideref () at pp_hot.c​:2346
#19 0x000000000055a245 in Perl_runops_debug () at dump.c​:2239
#20 0x00000000004623d3 in S_run_body (oldscope=1) at perl.c​:2517
#21 0x00000000004619fe in perl_run (my_perl=0xa9c010) at perl.c​:2440
#22 0x000000000041eae0 in main (argc=5, argv=0x7fffffffe608, env=0x7fffffffe638)
  at perlmain.c​:116
(gdb) f 4
#4 0x0000000000618df3 in S_find_uninit_var (obase=0xabc100, uninit_sv=0xa9e048, match=false,
  desc_p=0x7fffffffd518) at sv.c​:16093
16093 assert(is_hv); /* AV index is an IV and can't be undef */
(gdb) l
16088 if (index_gv)
16089 return varname(index_gv, '$', 0, NULL, 0, FUV_SUBSCRIPT_NONE);
16090 if (index_targ)
16091 return varname(NULL, '$', index_targ,
16092 NULL, 0, FUV_SUBSCRIPT_NONE);
16093 assert(is_hv); /* AV index is an IV and can't be undef */
16094 /* can a const HV index ever be undef? */
16095 return NULL;
16096 }
16097
(gdb) info locals
agg_targ = 0
index_type = 16
items = 0xaacbf0
last = 0xaacbd8
is_hv = false
agg_gv = 0x0
index_targ = 0
index_gv = 0x0
index_const_iv = 0
actions = 84
index_const_sv = 0x0
depth = 2
sv = 0x55ba9e <Perl_safesysrealloc+131>
gv = 0x48
o = 0xaacb80
o2 = 0x1a
kid = 0xaacb80
__PRETTY_FUNCTION__ = "S_find_uninit_var"

**VALGRIND**

No reported memory management errors.

**PERL -V**

dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 25 subversion 2) configuration​:
  Commit id​: c29dfc6
  Platform​:
  osname=linux, osvers=4.5.0-2-amd64, archname=x86_64-linux-ld
  uname='linux nightshade64 4.5.0-2-amd64 #1 smp debian 4.5.3-2 (2016-05-08) x86_64 gnulinux '
  config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache gcc-6.1 -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -DDEBUGGING -DPERL_POISON -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='ccache gcc-6.1', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='6.1.0', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='ccache gcc-6.1', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/local/lib/gcc/x86_64-pc-linux-gnu/6.1.0/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.22.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.22'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
  USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES
  USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
  USE_PERLIO USE_PERL_ATOF
  Built under linux
  Compiled at May 26 2016 17​:57​:37
  @​INC​:
  lib
  /usr/local/perl-afl/lib/site_perl/5.25.2/x86_64-linux-ld
  /usr/local/perl-afl/lib/site_perl/5.25.2
  /usr/local/perl-afl/lib/5.25.2/x86_64-linux-ld
  /usr/local/perl-afl/lib/5.25.2
  /usr/local/perl-afl/lib/site_perl/5.25.1
  /usr/local/perl-afl/lib/site_perl/5.24.0
  /usr/local/perl-afl/lib/site_perl
  .

[p5pRT]

From @iabyn

On Thu, May 26, 2016 at 05​:08​:31PM -0700, Dan Collins wrote​:

dcollins@​nightshade64​:~/perldebug$ ./perl -Ilib -W -e '$ISA[0][0]'
Useless use of array element in void context at -e line 1.
perl​: sv.c​:16093​: S_find_uninit_var​: Assertion `is_hv' failed.
Aborted

Fixed by the below​:

commit 6fe82bd
Author​: David Mitchell <davem@​iabyn.com>
AuthorDate​: Thu Jul 7 16​:24​:41 2016 +0100
Commit​: David Mitchell <davem@​iabyn.com>
CommitDate​: Thu Jul 7 16​:24​:41 2016 +0100

  handle magic in multideref "unit val" var names
 
  [perl #128253] Assert fail in S_find_uninit_var
 
  $ perl5240 -we'$ISA[0][0]'
  Useless use of array element in void context at -e line 1.
  perl5240​: sv.c​:16078​: S_find_uninit_var​: Assertion `is_hv' failed.
 
  The code in find_uninit_var() which looks for a variable name associated
  with an uninitialized value assumed, in the OP_MULTIDEREF branch, that the
  value was either an index if the op was top-level ($foo[$uninit]), or an
  array/hash element otherwise (1+$foo[...]).
 
  It turns out here's a third possibility​: magic. In $ISA[0][0], the first
  array lookup is in lval context, so it initially autovivifies to undef.
  Normally it would shortly afterwards be upgraded to a ref to an empty AV,
  but first ISA set magic for @​ISA is invoked. This ends up scanning @​ISA
  and finds an uninit value which it tries to use as a key into the stash
  cache, triggering an ununit value warning.
 
  This commit expands the OP_MULTIDEREF code in find_uninit_var() to handle
  this third possibility - chiefly by not returning a variable name unless
  the index var is the same SV as the uninit value. As well as fixing the
  assert failure in this ticket, it also stops printing an incorrect index in
  code like this​:
 
  $ perl -we'my $i = 0; $ISA[$i] = 1'
 
  before​:
 
  ....
  Use of uninitialized value $i in array element at -e line 1.
  Use of uninitialized value $i in array element at -e line 1.
  Recursive inheritance detected in package 'main' at -e line 1.
 
  after​:
 
  ....
  Use of uninitialized value in array element at -e line 1.
  Use of uninitialized value in array element at -e line 1.
  Recursive inheritance detected in package 'main' at -e line 1.
 
  @​ISA magic still has recursion issues with undef values, as can be seen
  above. I don't address those issues here. Because of that, I haven't
  been able to add tests.

--
Any [programming] language that doesn't occasionally surprise the
novice will pay for it by continually surprising the expert.
  -- Larry Wall

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

@iabyn - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'