segmentation fault Running binmode(STDOUT, ":encoding(utf8)") with ithreads

[p5pRT]

Migrated from rt.perl.org#41121 (status was 'resolved')

Searchable as RT41121$

[p5pRT]

From imacat@mail.imacat.idv.tw

Created by imacat@mail.imacat.idv.tw

  Hi. This is imacat from Taiwan. Perl crashed when spawning threads
while STDOUT/STDERR is wrapped with the :encoding(xxx) output layer.
Strangely it works find if the output layer is :utf8, but crashed
if :encoding(utf8). This cause me a lot of trouble since my terminal
is Big5 and I can only use :encoding(big5).

  The demostration is below. Please tell me if you need any information.

imacat@​rinse ~ % perl -mthreads -e'binmode STDOUT, "​:utf8"; threads->new(sub { })->join;'
imacat@​rinse ~ % perl -mthreads -e'binmode STDOUT, "​:encoding(utf8)"; threads->new(sub { })->join;'
zsh​: segmentation fault perl -mthreads
imacat@​rinse ~ % perl -mthreads -e'binmode STDOUT, "​:encoding(big5)"; threads->new(sub { })->join;'
zsh​: segmentation fault perl -mthreads
imacat@​rinse ~ %

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.8:

Configured by imacat at Tue May  9 09:15:58 CST 2006.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.16.14, archname=x86_64-linux-thread-multi-ld
    uname='linux yuki 2.6.16.14 #1 smp mon may 8 23:44:32 cst 2006 x86_64 gnulinux '
    config_args='-s -d -Dusethreads -Dcc=gcc -Duselongdouble -Doptimize=-g -O3 -Duse64bitint -Duse64bitall -Dprefix=/usr -Dd_dosuid -Dotherlibdirs=/usr/share/perl5 -Dinc_version_list=none -Acccdlflags=-fPIC -Duseshrplib=true -Dcf_email=imacat@mail.imacat.idv.tw'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=define
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-g -O3',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include'
    ccversion='', gccversion='3.4.4 20050314 (prerelease) (Debian 3.4.3-13)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
    alignbytes=16, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/x86_64-linux-thread-multi-ld/CORE'
    cccdlflags=' -fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.8:
    /home/imacat/lib/perl5
    /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi-ld
    /usr/lib/perl5/5.8.8
    /usr/lib/perl5/site_perl/5.8.8/x86_64-linux-thread-multi-ld
    /usr/lib/perl5/site_perl/5.8.8
    /usr/lib/perl5/site_perl
    /usr/share/perl5
    .


Environment for perl v5.8.8:
    HOME=/home/imacat
    LANG=zh_TW
    LANGUAGE=zh_TW
    LC_COLLATE=zh_TW
    LC_CTYPE=zh_TW
    LC_MESSAGES=zh_TW
    LC_MONETARY=zh_TW
    LC_NUMERIC=zh_TW
    LC_TIME=zh_TW
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/imacat/bin:/bin:/usr/bin:/opt/java/bin:/usr/local/bin
    PERL5LIB=/home/imacat/lib/perl5
    PERL5_CPANPLUS_CONFIG=/home/imacat/.cpanplus/config
    PERL_BADLANG (unset)
    SHELL=/bin/zsh

[p5pRT]

From ulfer.dm@gmail.com

Created by alexdm@lwd.ru

When Perl starts another interpreter thread, it segfaults
when trying to 'use' package declared with
'use encoding 'utf8';'. Imagine these two
files​:

bug.pl​:
  use threads;
  use check_utf8;
  my $t1 = async { };

check_utf8.pm​:
  package check_utf8;
  use encoding 'utf8';
  1;

It doesn't matter whether any code is called in async { }
or after it. Direct 'use' of check_utf8 in async { }
also causes segfault​:
  use threads;
  async { use check_utf8; }

However, if you write bug.pl like this​:
  use threads;
  my $t1 = async {
  eval "use check_utf8;";
  # call subroutines from check_utf8;
  # ...
  }

it will work perfectly. I should notice that
  eval { use check_utf8; }
will not work and will segfault. Similarly, any setups
where check_utf8 is included in main thread (and another
is started after inclusion) cause segfault​:
  use threads;
  eval "use check_utf8;";
  async { };

This bug was discovered in ActivePerl under windows 2000
and was reproduced under Linux. So, I think it is a core
bug. I can't use only use utf8; because my code contains
some messages in Russian, and those 'Wide character in
print...' should be avoided.

wbr, Alexey.

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.8:

Configured by Debian Project at Mon Nov 12 06:19:22 UTC 2007.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.22.10, archname=i486-linux-gnu-thread-multi
    uname='linux ninsei 2.6.22.10 #1 smp preempt thu oct 25 08:49:01 pdt
2007 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.8.8
-Dsitearch=/usr/local/lib/perl/5.8.8 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio
-Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN
-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.2.3 20071014 (prerelease) (Debian
4.2.2-3)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.6.1.so, so=so, useshrplib=true,
libperl=libperl.so.5.8.8
    gnulibc_version='2.6.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:

@INC for perl v5.8.8:
    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.8:
    HOME=/home/ulfer
    LANG=ru_RU.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From alex.davies@talktalk.net

Created by alex@AMELIE

The following crashes with 5.8, 5.10 & blead​:

# %<
use warnings;
use strict;
use threads;

my $BUG = 1;

binmode STDOUT, "​:encoding(utf8)" or die
  if $BUG;

if (0 and $^O eq 'MSWin32') {
  print STDERR "fork'ing on Win32\n";
  if (fork() == 0) {
  doit();
  }
} else {
  print STDERR "thread'ing\n";
  my $thrd = threads->create(\&doit);
  $thrd->join();
}

sub doit {
  binmode STDOUT, "​:encoding(utf8)" or die
  unless $BUG;

  my $f = $0; # arbitrary file
  open F, "<", $f or die;
  print <F>;
  close F;
}
# >%

If BUG is set to false ie. STDOUT is encoded in the thread,
then it works as expected. On Win32 it fails with both
threaded and forked execution.

It crashes in PerlIOEncode_getarg() in encoding.xs
at the PUSHSTACKi. The problem is that
my_perl->Icurstackinfo is NULL.

The point where it crashes is beneath

sv.c​:11887​: PerlIO_clone(aTHX_ proto_perl, param);

in perl_clone() but PL_curstackinfo is not initialised
until a later point​:

sv.c​:12240​: PL_curstackinfo = si_dup(proto_perl->Icurstackinfo, param);

So i guess the order of the initializations needs shuffling...
although that looks like a game of kerplunk :-)

Cheers,
alex.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.10.0:

Configured by adam at Mon Aug 11 04:41:10 2008.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags =' -s -O2 -DWIN32 -DHAVE_DES_FCRYPT  -DPERL_IMPLICIT_CONTEXT -

DPERL_IMPLICIT_SYS -fno-strict-aliasing -DPERL_MSVCRT_READFIX',
    optimize='-s -O2',
    cppflags='-DWIN32'
    ccversion='', gccversion='3.4.5', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'
    libpth=C:\strawberry\c\lib
    libs= -lmsvcrt -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -

ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion

-lodbc32 -lodbccp32
    perllibs= -lmsvcrt -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -

ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion

-lodbc32 -lodbccp32
    libc=-lmsvcrt, so=dll, useshrplib=true, libperl=libperl510.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE" -

L"C:\strawberry\c\lib"'

Locally applied patches:



@INC for perl 5.10.0:
    C:/strawberry/perl/lib
    C:/strawberry/perl/site/lib
    .


Environment for perl 5.10.0:
    HOME=C:\alex
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\alex\bin;C:\Program 
Files\Microsoft SDK\Bin\.;C:\Program Files\Microsoft  SDK\Bin\WinNT\.;C:\Program 
Files\Tcl-8.5.0\bin;C:\PerlAPPv9\bin;C:\strawberry\c\bin;C:\strawberry\perl\bin;C:\Program 
Files\Git\cmd;C:\Program
Files\Microsoft Visual Studio\VC98\Bin;C:\Program Files\Microsoft SDK\Bin\.;C:\Program 
Files\Microsoft SDK\Bin\WinNT;C:\cygwin\bin
    PERL_BADLANG (unset)
    SHELL (unset)

[p5pRT]

@jkeenan - Status changed from 'new' to 'open'

[p5pRT]

From @sciurius

Perl 5.18.0​:

% perl -e 'use
threads;binmode(STDOUT,"​:encoding(utf8)");threads->create(sub{})'
Segmentation fault (core dumped)

But​:

% perl -e 'use threads;binmode(STDOUT,"​:utf8");threads->create(sub{})'
Perl exited with active threads​:
  0 running and unjoined
  1 finished and unjoined
  0 running and detached

... 6 years, already ...

[p5pRT]

From @dcollinsn

Fixed by​:

0ee3fa2 is the first new commit
commit 0ee3fa2
Author​: Vincent Pit <perl@​profvince.com>
Date​: Fri Aug 28 14​:17​:00 2015 -0300

  Properly duplicate PerlIO​::encoding objects

  PerlIO​::encoding objects are usually initialized by calling Perl methods,
  essentially from the pushed() and getarg() callbacks. During cloning, the
  PerlIO API will by default call these methods to initialize the duplicate
  struct when the PerlIOBase parent struct is itself duplicated. This does
  not behave so well because the perl interpreter is not ready to call
  methods at this point, for the stacks are not set up yet.

  The proper way to duplicate the PerlIO​::encoding object is to call sv_dup()
  on its members from the dup() PerlIO callback. So the only catch is to make
  the getarg() and pushed() calls implied by the duplication of the underlying
  PerlIOBase object aware that they are called during cloning, and make them
  wait that the control flow returns to the dup() callback. Fortunately,
  getarg() knows since its param argument is then non-null, and its return
  value is passed immediately to pushed(), so it is enough to tag this
  returned value with a custom magic so that pushed() can see it is being
  called during cloning.

  This fixes [RT #31923].

:100644 100644 f79df008098f4df1f97f7f104171b89c31049a90 34c2e2436a2a14e776725ca5c40a74c8f0a4bdb7 M MANIFEST
:040000 040000 ce197fb6173fb13198e80fa4a743778f88a797d4 16376953f016ecdaf6dbe144a432d6d683e00229 M ext

5.22 is still broken, needs to be backported to next maint release? Testing my test now.

--
Respectfully,
Dan Collins

[p5pRT]

From @cpansprout

On Thu Jul 14 13​:25​:12 2016, dcollinsn@​gmail.com wrote​:

Fixed by​:

0ee3fa2 is the first new commit
commit 0ee3fa2
Author​: Vincent Pit <perl@​profvince.com>
Date​: Fri Aug 28 14​:17​:00 2015 -0300

Properly duplicate PerlIO​::encoding objects

PerlIO​::encoding objects are usually initialized by calling Perl
methods,
essentially from the pushed() and getarg() callbacks. During cloning,
the
PerlIO API will by default call these methods to initialize the
duplicate
struct when the PerlIOBase parent struct is itself duplicated. This
does
not behave so well because the perl interpreter is not ready to call
methods at this point, for the stacks are not set up yet.

The proper way to duplicate the PerlIO​::encoding object is to call
sv_dup()
on its members from the dup() PerlIO callback. So the only catch is to
make
the getarg() and pushed() calls implied by the duplication of the
underlying
PerlIOBase object aware that they are called during cloning, and make
them
wait that the control flow returns to the dup() callback. Fortunately,
getarg() knows since its param argument is then non-null, and its
return
value is passed immediately to pushed(), so it is enough to tag this
returned value with a custom magic so that pushed() can see it is
being
called during cloning.

This fixes [RT #31923].

:100644 100644 f79df008098f4df1f97f7f104171b89c31049a90
34c2e2436a2a14e776725ca5c40a74c8f0a4bdb7 M MANIFEST
:040000 040000 ce197fb6173fb13198e80fa4a743778f88a797d4
16376953f016ecdaf6dbe144a432d6d683e00229 M ext

5.22 is still broken, needs to be backported to next maint release?
Testing my test now.

Did this also fix #85366?

--

Father Chrysostomos

[p5pRT]

From @dcollinsn

Test added in 68ca57d, closing.

--
Respectfully,
Dan Collins

[p5pRT]

From [Unknown Contact. See original ticket]

Test added in 68ca57d, closing.

--
Respectfully,
Dan Collins

[p5pRT]

@dcollinsn - Status changed from 'open' to 'resolved'