glob in threads is broken in perl5.18

[p5pRT]

Migrated from rt.perl.org#119897 (status was 'resolved')

Searchable as RT119897$

[p5pRT]

From trast@inf.ethz.ch

It seems <glob*> contains a race when used in parallel in threads. This
test case reliably spews a few errors, and then segfaults, when I run it
on perl v5.16.2 (stock opensuse 12.3)​:

----- 8< -----
#!/usr/bin/perl

use warnings;
use strict;

use threads;
use threads​::shared;

my $nthread = 20;

sub work {
  foreach (1..10000) {
  my @​files = <*.supp>;
  }
}

my @​threads;
foreach my $i (1..$nthread) {
  push @​threads, threads->create(\&work);
}

foreach my $t (@​threads) {
  $t->join();
}
----- >8 -----

There are two kinds of errors shown​:

a) Attempt to free unreferenced scalar​: SV 0xd1e738, Perl interpreter​: 0xdf8900 at /home/thomas/tmp/perl-thread-glob-test.pl line 13.

b) Thread 1 terminated abnormally​: sv_upgrade from type 255 down to type 11 at perl-thread-glob-test.pl line 13.

Followed by a segmentation fault or (rarely) a bus error.

It does not matter whether the glob actually matches any files, but
running in a directory where it does match something greatly increases
the frequency of (a) errors, whereas running in a directory where it
doesn't greatly increases the frequency of (b) errors.

The segfault looks like this​:

  #0 0x0000000000499720 in Perl_hv_common (my_perl=0x8f7e80, hv=0x889720, keysv=keysv@​entry=0x0,
  key=key@​entry=0x7fffe8001630 "0", klen=<optimized out>, flags=0, action=action@​entry=48,
  val=val@​entry=0x0, hash=1849449579, hash@​entry=0) at hv.c​:638
  #1 0x000000000049b57e in Perl_hv_common_key_len (my_perl=my_perl@​entry=0x8f7e80, hv=<optimized out>,
  key=key@​entry=0x7fffe8001630 "0", klen_i32=<optimized out>, action=action@​entry=48,
  val=val@​entry=0x0, hash=hash@​entry=0) at hv.c​:333
  #2 0x00007ffff68b55d5 in iterate (my_perl=0x8f7e80, globber=0x7ffff68b5c60 <csh_glob>) at Glob.xs​:84
  #3 0x00000000004e7225 in Perl_pp_glob (my_perl=0x8f7e80) at pp_sys.c​:379
  #4 0x000000000049ee16 in Perl_runops_standard (my_perl=0x8f7e80) at run.c​:41
  #5 0x0000000000436deb in Perl_call_sv (my_perl=0x8f7e80, sv=0x9baba0, flags=11) at perl.c​:2705
  #6 0x00007ffff6ece0c9 in ?? ()
  from /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/auto/threads/threads.so
  #7 0x00007ffff7488e0f in start_thread () from /lib64/libpthread.so.0
  #8 0x00007ffff71bc7dd in clone () from /lib64/libc.so.6

or sometimes just starts at the iterate() frame​:

  #0 0x00007ffff68b569b in iterate (my_perl=0xcbad60, globber=0x7ffff68b5c60 <csh_glob>) at Glob.xs​:96
  #1 0x00000000004e7225 in Perl_pp_glob (my_perl=0xcbad60) at pp_sys.c​:379
  #2 0x000000000049ee16 in Perl_runops_standard (my_perl=0xcbad60) at run.c​:41
  #3 0x0000000000436deb in Perl_call_sv (my_perl=0xcbad60, sv=0xd7c690, flags=11) at perl.c​:2705
  #4 0x00007ffff6ece0c9 in ?? ()
  from /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/auto/threads/threads.so
  #5 0x00007ffff7488e0f in start_thread () from /lib64/libpthread.so.0
  #6 0x00007ffff71bc7dd in clone () from /lib64/libc.so.6

vincent on #perl ran the test script on many perl versions (thanks!) and
reported this result​:

  $ for p in ~/perl/builds/bin/perl5*thr* ; do echo $p ; $p x.pl ; done
  /home/vince/perl/builds/bin/perl5.10.0-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.10.1-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.12.0-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.12.1-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.12.2-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.12.3-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.12.4-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.12.5-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.14.0-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.14.1-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.14.2-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.14.3-dbg-psn-shr-thr-64
  /home/vince/perl/builds/bin/perl5.16.0-dbg-psn-shr-thr-64
  perl5.16.0-dbg-psn-shr-thr-64​: Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff)) != SVt_PVAV' failed.
  Aborted
  /home/vince/perl/builds/bin/perl5.16.1-dbg-psn-shr-thr-64
  perl5.16.1-dbg-psn-shr-thr-64​: Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff)) != SVt_PVAV' failed.
  Aborted
  /home/vince/perl/builds/bin/perl5.16.2-dbg-psn-shr-thr-64
  perl5.16.2-dbg-psn-shr-thr-64​: Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff)) != SVt_PVAV' failed.
  perl5.16.2-dbg-psn-shr-thr-64​: Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff)) != SVt_PVAV' failed.
  Aborted
  /home/vince/perl/builds/bin/perl5.17.3-dbg-psn-shr-thr-64
  perl5.17.3-dbg-psn-shr-thr-64​: Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff)) != SVt_PVAV' failed.
  Aborted
  /home/vince/perl/builds/bin/perl5.17.6-dbg-psn-shr-thr-64
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.17.7-dbg-psn-shr-thr-64
  Attempt to free nonexistent shared string ' ��', Perl interpreter​: 0x1a4c5c0 at x.pl line 13.
  Segmentation fault
  /home/vince/perl/builds/bin/perl5.8.0-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.1-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.2-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.3-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.4-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.5-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.6-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.7-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.8-dbg-shr-thr-64
  /home/vince/perl/builds/bin/perl5.8.9-dbg-psn-shr-thr-64

--- perlbug -d ---

---

Flags​:
  category=core
  severity=low

---

This perlbug was built using Perl 5.16.2 - Mon Mar 11 11​:01​:56 UTC 2013
It is being executed now by Perl 5.16.2 - Mon Mar 11 10​:54​:42 UTC 2013.

Site configuration information for perl 5.16.2​:

Configured by abuild at Mon Mar 11 10​:54​:42 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 2) configuration​:
 
  Platform​:
  osname=linux, osvers=3.4.6-2.10-xen, archname=x86_64-linux-thread-multi
  uname='linux build05 3.4.6-2.10-xen #1 smp thu jul 26 09​:36​:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux '
  config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
  ccversion='', gccversion='4.7.2 20130108 [gcc-4_7-branch revision 195012]', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
  libpth=/lib64 /usr/lib64 /usr/local/lib64
  libs=-lm -ldl -lcrypt -lpthread
  perllibs=-lm -ldl -lcrypt -lpthread
  libc=/lib64/libc-2.17.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.17'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches​:
 

---

@​INC for perl 5.16.2​:
  /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
  /usr/lib/perl5/site_perl/5.16.2
  /usr/lib/perl5/vendor_perl/5.16.2/x86_64-linux-thread-multi
  /usr/lib/perl5/vendor_perl/5.16.2
  /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi
  /usr/lib/perl5/5.16.2
  /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
  /usr/lib/perl5/site_perl/5.16.2
  /usr/lib/perl5/site_perl
  .

---

Environment for perl 5.16.2​:
  HOME=/home/thomas
  LANG=en_US.UTF-8
  LANGUAGE=
  LC_NUMERIC=POSIX
  LD_LIBRARY_PATH=/usr/lib64/mpi/gcc/openmpi/lib64​:/home/thomas/.local/lib​:/home/thomas/.local/lib64​:/home/thomas/.local/lib64/nest
  LOGDIR (unset)
  PATH=/home/thomas/.local/bin​:/usr/lib64/mpi/gcc/openmpi/bin​:/home/thomas/bin​:/usr/local/bin​:/usr/bin​:/bin​:/usr/bin/X11​:/usr/X11R6/bin​:/usr/games​:/home/thomas/.local/bin​:/usr/local/sbin​:/usr/sbin​:/sbin​:/usr/games
  PERL_BADLANG (unset)
  SHELL=/bin/bash

--
Thomas Rast
trast@​{inf,student}.ethz.ch

[p5pRT]

From @cpansprout

On Wed May 01 04​:55​:31 2013, trast@​inf.ethz.ch wrote​:

It seems <glob*> contains a race when used in parallel in threads.
This
test case reliably spews a few errors, and then segfaults, when I run
it
on perl v5.16.2 (stock opensuse 12.3)​:

----- 8< -----
#!/usr/bin/perl

use warnings;
use strict;

use threads;
use threads​::shared;

my $nthread = 20;

sub work {
foreach (1..10000) {
my @​files = <*.supp>;
}
}

my @​threads;
foreach my $i (1..$nthread) {
push @​threads, threads->create(\&work);
}

foreach my $t (@​threads) {
$t->join();
}
----- >8 -----

There are two kinds of errors shown​:

a) Attempt to free unreferenced scalar​: SV 0xd1e738, Perl interpreter​:
0xdf8900 at /home/thomas/tmp/perl-thread-glob-test.pl line 13.

b) Thread 1 terminated abnormally​: sv_upgrade from type 255 down to
type 11 at perl-thread-glob-test.pl line 13.

Followed by a segmentation fault or (rarely) a bus error.

I can produce this bug on Mac OS X as well.

I also see ‘Attempt to free nonexistent shared string’, ‘Assertion
failed’, ‘panic​: free from wrong pool’. The errors I get are different
in each run.

I have tried to diagnose this, but I am stumped.

It has nothing to do with readdir. I get the same thing with <{a}>, but
<{a,b}{a,b}> triggers it more reliably.

The bug can’t be in glob_expand. I changed glob_expand to return early,
and the memory corruption persists.

But I can’t see anything wrong with the code. Nothing uses globals.

--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @cpansprout

On Sat Jul 06 17​:53​:30 2013, sprout wrote​:

The bug can’t be in glob_expand.

I meant globextend.

--

Father Chrysostomos

[p5pRT]

From @nthykier

Created by @nthykier

Dear perl porters,

When Debian migrated to Perl5.18.1, I started to experience random
seg. faults in one of my perl scripts. hugmeir from #p5p devised
the following mimimal test case​:

"""
use threads;
scalar glob("*");
threads->create(sub { glob("*") })->join();
"""

The test case itself does not crash, but it does spew out warnings
like​:

"""
Unbalanced string table refcount​: (1) for "I" during global destruction.
Attempt to free nonexistent shared string 'I', [...]
Attempt to free unreferenced scalar​: SV 0x8c6a1fc, [...]
"""

Based on that, I replaced my calls to "glob" in the threads and my script
stopped crashing.

hugmeir also suggested that he believed this bug to also affect blead
(I have not tested that assertion). TonyC from #p5p suggested that it
might be because the "glob state isn't being cloned".

~Niels

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.18.1:

Configured by Debian Project at Sun Sep  1 00:18:57 UTC 2013.

Summary of my perl5 (revision 5 version 18 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=3.2.0-4-amd64, archname=i486-linux-gnu-thread-multi-64int
    uname='linux binet 3.2.0-4-amd64 #1 smp debian 3.2.46-1+deb7u1 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.18 -Darchlib=/usr/lib/perl/5.18 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.18.1 -Dsitearch=/usr/local/lib/perl/5.18.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.18.1 -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.8.1', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/i386-linux-gnu /lib/../lib /usr/lib/i386-linux-gnu /usr/lib/../lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=, so=so, useshrplib=true, libperl=libperl.so.5.18.1
    gnulibc_version='2.17'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - http://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - http://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - http://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: Set location of libperl.a to /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/prefix_changes - Fiddle with *PREFIX and variables written to the makefile
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/module_build_man_extensions - http://bugs.debian.org/479460 Adjust Module::Build manual page extensions for the Debian Perl policy
    DEBPKG:debian/prune_libs - http://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] http://bugs.debian.org/100195 Document the Net::SMTP 'Port' option
    DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/cpanplus_definstalldirs - http://bugs.debian.org/533707 Configure CPANPLUS to use the site directories by default.
    DEBPKG:debian/cpanplus_config_path - Save local versions of CPANPLUS::Config::System into /etc/perl.
    DEBPKG:debian/deprecate-with-apt - http://bugs.debian.org/702096 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
    DEBPKG:debian/patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.18.1-3 in patchlevel.h
    DEBPKG:debian/skip-kfreebsd-crash - http://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG:fixes/document_makemaker_ccflags - http://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - http://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/hurd_test_todo_syslog - http://bugs.debian.org/650093 Disable failing GNU/Hurd tests in cpan/Sys-Syslog/t/syslog.t
    DEBPKG:debian/hurd_test_skip_sigdispatch - http://bugs.debian.org/650188 Disable failing GNU/Hurd tests op/sigdispatch.t
    DEBPKG:debian/hurd_test_skip_stack - http://bugs.debian.org/650175 Disable failing GNU/Hurd tests dist/threads/t/stack.t
    DEBPKG:debian/hurd_test_skip_pipe - http://bugs.debian.org/650187 Disable failing GNU/Hurd tests io/pipe.t
    DEBPKG:debian/hurd_test_skip_io_pipe - http://bugs.debian.org/650096 Disable failing GNU/Hurd tests dist/IO/t/io_pipe.t
    DEBPKG:fixes/manpage_name_Test-Harness - http://bugs.debian.org/650451 [rt.cpan.org #73399] cpan/Test-Harness: add NAME headings in modules with POD
    DEBPKG:debian/makemaker-pasthru - http://bugs.debian.org/660195 [rt.cpan.org #28632] Make EU::MM pass LD through to recursive Makefile.PL invocations
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - http://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - http://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:debian/hurd_net_ping_disable_test - http://bugs.debian.org/709385 Disable failing Net-Ping tests for GNU/Hurd
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] http://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:fixes/net_ftp_failed_command - [rt.cpan.org #37700] http://bugs.debian.org/491062 Net::FTP: cope gracefully with a failed command
    DEBPKG:fixes/perlbug-patchlist - [3541c11] http://bugs.debian.org/710842 [perl #118433] Make perlbug look up the list of local patches at run time
    DEBPKG:fixes/regexp-preserve - http://bugs.debian.org/718209 [perl #118213] [f4194b2] RT #118213: handle $r=qr/.../; /$r/p properly
    DEBPKG:fixes/regexp-preserve-testcases - http://bugs.debian.org/718209 [perl #118213] [4d7b2f5] Disable new //p tests
    DEBPKG:fixes/module_metadata_security_doc - [68cdd4b] CVE-2013-1437 documentation fix


@INC for perl 5.18.1:
    /etc/perl
    /usr/local/lib/perl/5.18.1
    /usr/local/share/perl/5.18.1
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.18
    /usr/share/perl/5.18
    /usr/local/lib/site_perl
    .


Environment for perl 5.18.1:
    LANG=en_DK.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @cpansprout

On Fri Sep 20 04​:09​:25 2013, niels@​thykier.net wrote​:

This is a bug report for perl from niels@​thykier.net,
generated with the help of perlbug 1.39 running under perl 5.18.1.

-----------------------------------------------------------------
[Please describe your issue here]

Dear perl porters,

When Debian migrated to Perl5.18.1, I started to experience random
seg. faults in one of my perl scripts. hugmeir from #p5p devised
the following mimimal test case​:

"""
use threads;
scalar glob("*");
threads->create(sub { glob("*") })->join();
"""

The test case itself does not crash, but it does spew out warnings
like​:

"""
Unbalanced string table refcount​: (1) for "I" during global
destruction.
Attempt to free nonexistent shared string 'I', [...]
Attempt to free unreferenced scalar​: SV 0x8c6a1fc, [...]
"""

Based on that, I replaced my calls to "glob" in the threads and my
script
stopped crashing.

hugmeir also suggested that he believed this bug to also affect blead
(I have not tested that assertion). TonyC from #p5p suggested that it
might be because the "glob state isn't being cloned".

Interesting. Could this be the same as #117823?

--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @nthykier

On 2013-09-20 17​:39, Father Chrysostomos via RT wrote​:

[...]

Interesting. Could this be the same as #117823?

To be honest, I am not sure. When I discovered the issue, I ran a gdb
on it produce a stack trace (see attached file). From what I can tell,
the stack traces are different (mine involves "op-freeing" vs the
#117823 seems to deal with pp_glob)[0].
  That said, the symptoms and cause look similar[1]. So, I would not be
surprised if a fix for one of these bugs also fixes the other.

~Niels

[0] An extra data point is the assertion triggered. In the Debian bug
#723805[DebBug], Niko Tyni reported that he could trigger the following
assertion reliably.

debugperl​: hv.c​:356​: Perl_hv_common​: Assertion `((svtype)((hv)->sv_flags
& 0xff)) == SVt_PVHV' failed.

But RT#117823 lists​:

Glob.xs​:232​: csh_glob​: Assertion `((svtype)((entries)->sv_flags & 0xff))
!= SVt_PVAV' failed.

[1] E.g. both bugs involves calling "glob" from threads. My original
script also "rarely" (i.e. "once") triggered a bus error rather than a
seg. fault.

[DebBug] http​://bugs.debian.org/cgi-bin/bugreport.cgi?bug=723805

[p5pRT]

From @nthykier

gdb-bt.log

[p5pRT]

From @Hugmeir

On Fri, Sep 20, 2013 at 12​:39 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Sep 20 04​:09​:25 2013, niels@​thykier.net wrote​:

This is a bug report for perl from niels@​thykier.net,
generated with the help of perlbug 1.39 running under perl 5.18.1.

-----------------------------------------------------------------
[Please describe your issue here]

Dear perl porters,

When Debian migrated to Perl5.18.1, I started to experience random
seg. faults in one of my perl scripts. hugmeir from #p5p devised
the following mimimal test case​:

"""
use threads;
scalar glob("*");
threads->create(sub { glob("*") })->join();
"""

The test case itself does not crash, but it does spew out warnings
like​:

"""
Unbalanced string table refcount​: (1) for "I" during global
destruction.
Attempt to free nonexistent shared string 'I', [...]
Attempt to free unreferenced scalar​: SV 0x8c6a1fc, [...]
"""

Based on that, I replaced my calls to "glob" in the threads and my
script
stopped crashing.

hugmeir also suggested that he believed this bug to also affect blead
(I have not tested that assertion). TonyC from #p5p suggested that it
might be because the "glob state isn't being cloned".

Interesting. Could this be the same as #117823?

Yes. The problem for both is that x_GLOB_ENTRIES should be thread-local,
but is unintentionally being shared between threads.

This patch solves the issue​:

Inline Patch

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index b3705b3..181dbbc 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@@ -396,6 +396,13 @@ PPCODE:
     iterate(aTHX_ doglob_iter_wrapper);
     SPAGAIN;

+void
+CLONE(...)
+CODE:
+    PERL_UNUSED_ARG(items);
+    MY_CXT_CLONE;
+    MY_CXT.x_GLOB_ENTRIES = NULL;
+
 BOOT:
 {
 #ifndef PERL_EXTERNAL_GLOB

But it means that this:

./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} foo(); say
threads->create(\&foo)->join() for 1..3'

Will return "Artistic" four times, rather than "Artristic" followed by the
second file thrice. If the latter is the desired behavior, then we need to
hv_dup(x_GLOB_ENTRIES) in CLONE() as well.

[p5pRT]

From @cpansprout

On Fri Sep 20 16​:31​:00 2013, Hugmeir wrote​:

Yes. The problem for both is that x_GLOB_ENTRIES should be thread-local,
but is unintentionally being shared between threads.

This patch solves the issue​:

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index b3705b3..181dbbc 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@​@​ -396,6 +396,13 @​@​ PPCODE​:
iterate(aTHX_ doglob_iter_wrapper);
SPAGAIN;

+void
+CLONE(...)
+CODE​:
+ PERL_UNUSED_ARG(items);
+ MY_CXT_CLONE;
+ MY_CXT.x_GLOB_ENTRIES = NULL;
+
BOOT​:
{
#ifndef PERL_EXTERNAL_GLOB

But it means that this​:

./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} foo(); say
threads->create(\&foo)->join() for 1..3'

Will return "Artistic" four times, rather than "Artristic" followed by the
second file thrice. If the latter is the desired behavior, then we need to
hv_dup(x_GLOB_ENTRIES) in CLONE() as well.

Yes, the latter. That would be backward-compatible with when File​::Glob
still worked.

Thank you for tackling this. This is entirely my fault, as I was the
one who moved it from Perl space to XS.

--

Father Chrysostomos

[p5pRT]

From @Hugmeir

On Fri, Sep 20, 2013 at 9​:16 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Sep 20 16​:31​:00 2013, Hugmeir wrote​:

Yes. The problem for both is that x_GLOB_ENTRIES should be thread-local,
but is unintentionally being shared between threads.

This patch solves the issue​:

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index b3705b3..181dbbc 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@​@​ -396,6 +396,13 @​@​ PPCODE​:
iterate(aTHX_ doglob_iter_wrapper);
SPAGAIN;

+void
+CLONE(...)
+CODE​:
+ PERL_UNUSED_ARG(items);
+ MY_CXT_CLONE;
+ MY_CXT.x_GLOB_ENTRIES = NULL;
+
BOOT​:
{
#ifndef PERL_EXTERNAL_GLOB

But it means that this​:

./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} foo(); say
threads->create(\&foo)->join() for 1..3'

Will return "Artistic" four times, rather than "Artristic" followed by
the
second file thrice. If the latter is the desired behavior, then we need
to
hv_dup(x_GLOB_ENTRIES) in CLONE() as well.

Yes, the latter. That would be backward-compatible with when File​::Glob
still worked.

I'm afraid I don't know how to implement this properly! Unlike dup magic,
CLONE() doesn't get a CLONE_PARAMS argument, so calling sv_dup() and
friends requires manual intervention, and I don't believe that the correct
procedure is documented anywhere.
The only place in the core that does an sv_dup() during CLONE() is
threads.xs, and that's likely not a good role model for more normal code;
meanwhile, the paradigm in the few CPAN modules that do this[*] is "grab
the interpreter during BOOT, use that to construct a fake CLONE_PARAMS, and
then replace the interpreter with aTHX after MY_CXT_CLONE"
Which, on top of the previous patch, becomes this​:

Inline Patch

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index 181dbbc..118d88e 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@@ -9,6 +9,9 @@
 #define MY_CXT_KEY "File::Glob::_guts" XS_VERSION

 typedef struct {
+#ifdef USE_ITHREADS
+    tTHX interp;
+#endif
     int        x_GLOB_ERROR;
     HV *    x_GLOB_ENTRIES;
     Perl_ophook_t    x_GLOB_OLD_OPHOOK;
@@ -396,12 +399,32 @@ PPCODE:
     iterate(aTHX_ doglob_iter_wrapper);
     SPAGAIN;

+#ifdef USE_ITHREADS
+
 void
 CLONE(...)
+INIT:
+    HV *glob_entries_clone = NULL;
 CODE:
     PERL_UNUSED_ARG(items);
-    MY_CXT_CLONE;
-    MY_CXT.x_GLOB_ENTRIES = NULL;
+    {
+        dMY_CXT;
+        if ( MY_CXT.x_GLOB_ENTRIES ) {
+            CLONE_PARAMS param;
+            param.stashes    = NULL;
+            param.flags      = 0;
+            param.proto_perl = MY_CXT.interp;
+
+            glob_entries_clone =

MUTABLE_HV(sv_dup_inc((SV*)MY_CXT.x_GLOB_ENTRIES, ¶m)); \+ \} \+ \} \+ \{ \+ MY\_CXT\_CLONE; \+ MY\_CXT\.x\_GLOB\_ENTRIES = glob\_entries\_clone; \+ MY\_CXT\.interp = aTHX; \+ \} \+ \+\#endif

BOOT​:
{
@​@​ -418,6 +441,9 @​@​ BOOT​:
  dMY_CXT;
  MY_CXT.x_GLOB_ENTRIES = NULL;
  MY_CXT.x_GLOB_OLD_OPHOOK = PL_opfreehook;
+#ifdef USE_ITHREADS
+ MY_CXT.interp = aTHX;
+#endif
  PL_opfreehook = glob_ophook;
  }
}

And with that​:
./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} say foo(); say "\t",
threads->create(\&foo)->join() for 1..3; say ""; say "\t\t",
threads->create(sub { say "\t", foo(); threads->create(\&foo)->join()
})->join(); say foo()'
Artistic
  AUTHORS
  AUTHORS
  AUTHORS

  AUTHORS
  autodoc.pl
AUTHORS

So, it works, but no clue if this is the proper way of doing it.
Talking about proper ways and threads, I'm trying to write some tests for
this, but I've hit a bit of a conundrum. Test​::More requires threads to be
pre-loaded to work properly with threads -- but at the same time, I want to
skip all the tests if threads aren't available. I believe the code below is
the correct way of doing that, but it means that most code out there,
including bits in the core, is doing it wrong. Does that merit patching
those files?

use Config;
use if $Config{useithreads}, 'threads';
use Test​::More;

BEGIN {
  if (! $Config{'useithreads'}) {
  plan skip_all => "Perl not compiled with 'useithreads'";
  }
}

[*] At least, that's the impression that I got when I was looking some
weeks back, and what I ended up using in Params​::Lazy.

[p5pRT]

From @cpansprout

On Fri Sep 20 18​:05​:44 2013, Hugmeir wrote​:

On Fri, Sep 20, 2013 at 9​:16 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Sep 20 16​:31​:00 2013, Hugmeir wrote​:

Yes. The problem for both is that x_GLOB_ENTRIES should be
thread-local,
but is unintentionally being shared between threads.

This patch solves the issue​:

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index b3705b3..181dbbc 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@​@​ -396,6 +396,13 @​@​ PPCODE​:
iterate(aTHX_ doglob_iter_wrapper);
SPAGAIN;

+void
+CLONE(...)
+CODE​:
+ PERL_UNUSED_ARG(items);
+ MY_CXT_CLONE;
+ MY_CXT.x_GLOB_ENTRIES = NULL;
+
BOOT​:
{
#ifndef PERL_EXTERNAL_GLOB

But it means that this​:

./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} foo(); say
threads->create(\&foo)->join() for 1..3'

Will return "Artistic" four times, rather than "Artristic" followed by
the
second file thrice. If the latter is the desired behavior, then we
need
to
hv_dup(x_GLOB_ENTRIES) in CLONE() as well.

Yes, the latter. That would be backward-compatible with when File​::Glob
still worked.

I'm afraid I don't know how to implement this properly! Unlike dup magic,
CLONE() doesn't get a CLONE_PARAMS argument, so calling sv_dup() and
friends requires manual intervention, and I don't believe that the correct
procedure is documented anywhere.
The only place in the core that does an sv_dup() during CLONE() is
threads.xs, and that's likely not a good role model for more normal code;

Ah, but File​::Glob is *not* normal code!

meanwhile, the paradigm in the few CPAN modules that do this[*] is "grab
the interpreter during BOOT, use that to construct a fake
CLONE_PARAMS, and
then replace the interpreter with aTHX after MY_CXT_CLONE"
Which, on top of the previous patch, becomes this​:

It has often bothered me that a CLONE method in pure Perl can’t really
access both interpreters. I have known about CLONE for a long time, but
had never actually seen it used (or bothered looking).

If what you have below works, and works reliably (it looks fine to me),
I say go for it.

Is it possible to write this in a way that is binary-compatible (for
5.16 and 5.18)? I believe the fact that the struct below is not public
makes it safe to change it like that.

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index 181dbbc..118d88e 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@​@​ -9,6 +9,9 @​@​
#define MY_CXT_KEY "File​::Glob​::_guts" XS_VERSION

typedef struct {
+#ifdef USE_ITHREADS
+ tTHX interp;
+#endif
int x_GLOB_ERROR;
HV * x_GLOB_ENTRIES;
Perl_ophook_t x_GLOB_OLD_OPHOOK;
@​@​ -396,12 +399,32 @​@​ PPCODE​:
iterate(aTHX_ doglob_iter_wrapper);
SPAGAIN;

+#ifdef USE_ITHREADS
+
void
CLONE(...)
+INIT​:
+ HV *glob_entries_clone = NULL;
CODE​:
PERL_UNUSED_ARG(items);
- MY_CXT_CLONE;
- MY_CXT.x_GLOB_ENTRIES = NULL;
+ {
+ dMY_CXT;
+ if ( MY_CXT.x_GLOB_ENTRIES ) {
+ CLONE_PARAMS param;
+ param.stashes = NULL;
+ param.flags = 0;
+ param.proto_perl = MY_CXT.interp;
+
+ glob_entries_clone =
MUTABLE_HV(sv_dup_inc((SV*)MY_CXT.x_GLOB_ENTRIES, &param));
+ }
+ }
+ {
+ MY_CXT_CLONE;
+ MY_CXT.x_GLOB_ENTRIES = glob_entries_clone;
+ MY_CXT.interp = aTHX;
+ }
+
+#endif

BOOT​:
{
@​@​ -418,6 +441,9 @​@​ BOOT​:
dMY_CXT;
MY_CXT.x_GLOB_ENTRIES = NULL;
MY_CXT.x_GLOB_OLD_OPHOOK = PL_opfreehook;
+#ifdef USE_ITHREADS
+ MY_CXT.interp = aTHX;
+#endif
PL_opfreehook = glob_ophook;
}
}

And with that​:
./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} say foo(); say "\t",
threads->create(\&foo)->join() for 1..3; say ""; say "\t\t",
threads->create(sub { say "\t", foo(); threads->create(\&foo)->join()
})->join(); say foo()'
Artistic
AUTHORS
AUTHORS
AUTHORS

AUTHORS
    autodoc\.pl

AUTHORS

So, it works, but no clue if this is the proper way of doing it.
Talking about proper ways and threads, I'm trying to write some tests for
this, but I've hit a bit of a conundrum. Test​::More requires threads to be
pre-loaded to work properly with threads -- but at the same time, I
want to
skip all the tests if threads aren't available. I believe the code
below is
the correct way of doing that, but it means that most code out there,
including bits in the core, is doing it wrong. Does that merit patching
those files?

Probably.

use Config;
use if $Config{useithreads}, 'threads';
use Test​::More;

BEGIN {
if (! $Config{'useithreads'}) {
plan skip_all => "Perl not compiled with 'useithreads'";
}
}

[*] At least, that's the impression that I got when I was looking some
weeks back, and what I ended up using in Params​::Lazy.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Fri Sep 20 20​:19​:01 2013, sprout wrote​:

Is it possible to write this in a way that is binary-compatible (for
5.16 and 5.18)? I believe the fact that the struct below is not public
makes it safe to change it like that.

That gives the wrong impression. Let’s try again​:

I believe (and hope) the proposed changes are binary-compatible. If it
turns out I am wrong, please make them binary-compatible. Thank you.

--

Father Chrysostomos

[p5pRT]

From @Hugmeir

On Sat, Sep 21, 2013 at 12​:19 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Sep 20 18​:05​:44 2013, Hugmeir wrote​:

On Fri, Sep 20, 2013 at 9​:16 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Sep 20 16​:31​:00 2013, Hugmeir wrote​:

Yes. The problem for both is that x_GLOB_ENTRIES should be
thread-local,
but is unintentionally being shared between threads.

This patch solves the issue​:

diff --git a/ext/File-Glob/Glob.xs b/ext/File-Glob/Glob.xs
index b3705b3..181dbbc 100644
--- a/ext/File-Glob/Glob.xs
+++ b/ext/File-Glob/Glob.xs
@​@​ -396,6 +396,13 @​@​ PPCODE​:
iterate(aTHX_ doglob_iter_wrapper);
SPAGAIN;

+void
+CLONE(...)
+CODE​:
+ PERL_UNUSED_ARG(items);
+ MY_CXT_CLONE;
+ MY_CXT.x_GLOB_ENTRIES = NULL;
+
BOOT​:
{
#ifndef PERL_EXTERNAL_GLOB

But it means that this​:

./perl -Ilib -Mthreads -E 'sub foo {scalar glob("*")} foo(); say
threads->create(\&foo)->join() for 1..3'

Will return "Artistic" four times, rather than "Artristic" followed
by
the
second file thrice. If the latter is the desired behavior, then we
need
to
hv_dup(x_GLOB_ENTRIES) in CLONE() as well.

Yes, the latter. That would be backward-compatible with when
File​::Glob
still worked.

I'm afraid I don't know how to implement this properly! Unlike dup magic,
CLONE() doesn't get a CLONE_PARAMS argument, so calling sv_dup() and
friends requires manual intervention, and I don't believe that the
correct
procedure is documented anywhere.
The only place in the core that does an sv_dup() during CLONE() is
threads.xs, and that's likely not a good role model for more normal code;

Ah, but File​::Glob is *not* normal code!

Heh, fair enough.

meanwhile, the paradigm in the few CPAN modules that do this[*] is "grab
the interpreter during BOOT, use that to construct a fake
CLONE_PARAMS, and
then replace the interpreter with aTHX after MY_CXT_CLONE"
Which, on top of the previous patch, becomes this​:

It has often bothered me that a CLONE method in pure Perl can’t really
access both interpreters. I have known about CLONE for a long time, but
had never actually seen it used (or bothered looking).

If what you have below works, and works reliably (it looks fine to me),
I say go for it.

I wasn't sure if the new tests would work on VMS -- most of the other glob
test files have special cases for it -- so I've pushed this as
smoke-me/hugmeir/dup_glob_state

Is it possible to write this in a way that is binary-compatible (for
5.16 and 5.18)? I believe the fact that the struct below is not public
makes it safe to change it like that.

This I do not know.
(I can't find a definition of binary (in)?compatibility perlpolicy or other
pods)