Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO layer for STDERR not set #15392

p5pRT opened this issue Jun 10, 2016 · 4 comments

IO layer for STDERR not set #15392

p5pRT opened this issue Jun 10, 2016 · 4 comments


Copy link

p5pRT commented Jun 10, 2016

Migrated from (status was 'open')

Searchable as RT128365$

Copy link

p5pRT commented Jun 10, 2016

From @choroba

Created by @choroba

When reopening STDERR, its encoding layer doesn't change.

  use strict;
  use warnings;
  use utf8;

  open STDERR, '>​:utf8' , 'perl.log' or die $!;
  print STDERR "(é'-è_çà)\n";
  binmode STDERR, '​:utf8';
  print STDERR "(é'-è_çà)\n";

Output (contents of the perl.log file)​:


Expected output (works OK if I use LOG instead of STDERR, or my $LOG,
or close STDERR before opening)​:


Origin​: http​://


Perl Info


This perlbug was built using Perl 5.18.2 - Sun Oct 25 05:48:43 UTC 2015
It is being executed now by  Perl 5.18.2 - Sun Oct 25 05:46:02 UTC 2015.

Site configuration information for perl 5.18.2:

Configured by abuild at Sun Oct 25 05:46:02 UTC 2015.

Summary of my perl5 (revision 5 version 18 subversion 2) configuration:

     osname=linux, osvers=4.1.10-1-default, archname=x86_64-linux-thread-multi
     uname='linux cloud105 4.1.10-1-default #1 smp preempt mon oct 12 11:24:48 utc 2015 (67a24e6) x86_64 x86_64 x86_64 gnulinux '
     config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl -Dinc_version_list=5.18.0/x86_64-linux-thread-multi 5.18.0 5.18.1/x86_64-linux-thread-multi 5.18.1'
     hint=recommended, useposix=true, d_sigaction=define
     useithreads=define, usemultiplicity=define
     useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
     use64bitint=define, use64bitall=define, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
     cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
     optimize='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
     cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
     ccversion='', gccversion='4.8.5', gccosandvers=''
     intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
     ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
     libpth=/lib64 /usr/lib64 /usr/local/lib64
     libs=-lm -ldl -lcrypt -lpthread
     perllibs=-lm -ldl -lcrypt -lpthread
     libc=/lib64/, so=so, useshrplib=true,
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/CORE'
     cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:

@INC for perl 5.18.2:

Environment for perl 5.18.2:
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PERL_BADLANG (unset)
     PERL_MB_OPT=--install_base "/home/choroba/perl5"

Copy link

p5pRT commented Mar 27, 2017


E. Choroba wrote​:

When reopening STDERR, its encoding layer doesn't change.

This actually happens when, and only when, reopening a file handle that
currently has a fd in the std{in,out,err} range. It happens regardless
of which of those fds it is, and regardless of how the reopened handle
was originally opened. The code responsible is trying to preserve the
special fd across a reopen, which is behaviour that we want to keep.
It's certainly a bug that we don't honour the layer specification when
doing this. The code is in S_openn_cleanup() in doio.c, which has this
alarming comment​:

  /* Eeek - FIXME !!!
  * If this is a standard handle we discard all the layer stuff
  * and just dup the fd into whatever was on the handle before !

The issue is that the new handle configuration is initially opened on a
new fd and has to move to the special fd number, but (I think) we don't
have any API to change the fd number in an existing handle configuration.
The code is taking the easy, but wrong, approach of reusing the old
handle configuration, which of course already has the right fd number.


Copy link

p5pRT commented Mar 27, 2017

The RT System itself - Status changed from 'new' to 'open'

Copy link

p5pRT commented Mar 31, 2017


The problem of this ticket interacts nastily with threads.

Without threads, the same-fd-on-reopen semantic makes sense. Each file
handle owns its fd, and reopening a handle means we're finished using its
former open file. So upon reopen of a file handle with low-numbered fd,
that fd is free to be reused for the new open file. It would be easy
enough to fix the problem of this ticket​: we can add a perlio method to
swap around fd ownership, and thereby effectively move a file handle from
one fd number to another. (Only slightly complicated by the backcompat
need to support perlio layers that own a fd but don't implement this
new method.) Upon reopen we'd use this method and a bit of dupping to
put the old layer stack on a spare fd, put the new layer stack on the
low fd number, then close the old layer stack properly and leave the
new layer stack open with ownership of the low fd.

Threads complicate things because each file handle is duplicated into each
thread but they share the underlying fds. Thus a file handle doesn't
fully own its fd. There's a cross-thread refcounting arrangement,
which tracks how many handles are using a fd, and the fd is only closed
when there are no more handles using it. This works fine so long as fds
are significant only through their use by handles, such that we'd never
want to reuse a specific fd. It works fine with ordinary reopening​:
the new open file gets a fresh fd with refcount 1, the refcount of the
formerly shared fd is decremented, and the file handle that was reopened
no longer shares its fd with anything.

The desire to reuse a low fd number when reopening a handle conflicts
with fds being shared between threads. No matter how clean the treatment
of the reopened handle itself, the natural effect of fd reuse on other
handles that shared the fd is that they see the open file substituted
underneath their existing layer stack. That is, the other handles see
exactly the kind of unclean substitution of which this ticket complains.
Fixing the layer handling for reopening low fds in a single thread would
leave the cross-thread issue unsolved.

A similar issue arises with the ">&=" open mode. This sets up sharing
of a fd between two handles in a single thread.

The treatment of reopening with a shared fd is really a distinct bug from
the bug with which this ticket is concerned, but I think it is necessary
to determine its solution in order to properly tackle this ticket.
The shared case raises questions as to what reopening a file handle means,
and what we're actually trying to achieve with the reuse of low fds.

A relatively simple possible answer is that a fd can only be reused if
the handle being reopened is the only user of that fd. Reopening STDOUT
et al would work if done first thing in the main thread, but would fail
if attempted after multiple threads have been created which have kept the
STDOUT handle open. It should fail rather than get a fresh fd to avoid
the surprising behaviour of STDOUT ceasing to correspond to the standard
output that is passed to subprocesses. A downside is that if this reason
for reopen failure, that the fd is shared, must be expressed through the
narrow channel of errno, there is no errno value that expresses this well.
On the upside this is quite easy to implement​: the logic to check whether
a handle supports swapping fd ownership can readily say that it's not
swappable if it's not owned outright because it's shared.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

2 participants