Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perl hangs when two files are redirected into a pipe #953

Closed
p5pRT opened this issue Dec 15, 1999 · 5 comments
Closed

perl hangs when two files are redirected into a pipe #953

p5pRT opened this issue Dec 15, 1999 · 5 comments

Comments

@p5pRT
Copy link

p5pRT commented Dec 15, 1999

Migrated from rt.perl.org#1900 (status was 'resolved')

Searchable as RT1900$

@p5pRT
Copy link
Author

p5pRT commented Dec 15, 1999

From summer@OS2.ami.com.au

I took some code 'man perlfunc' and modified it to read thus​:
#!/usr/bin/perl
open(OLDOUT, ">&STDOUT");
open(OLDERR, ">&STDERR");

open(STDOUT, "| cat >foo.out") || die "Can't redirect stdout"; # JCS
open(STDERR, ">&STDOUT") || die "Can't dup stdout";

select(STDERR); $| = 1; # make unbuffered
select(STDOUT); $| = 1; # make unbuffered

print STDOUT "stdout 1\n"; # this works for
print STDERR "stderr 1\n"; # subprocesses too

close(STDOUT);
close(STDERR);

open(STDOUT, ">&OLDOUT");
open(STDERR, ">&OLDERR");

print STDOUT "stdout 2\n";
print STDERR "stderr 2\n";

The only line I changed is labeled JCS.

This hangs (waits indefinitely), apparently at one or other of the closes
(lines 14/15). I presume cat's not getting EOF.

I'm running perl on RHL 5.x​:
I'm not use I beleive this info;-)
[summer@​emu summer]$ perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 2) configuration​:
  Platform​:
  osname=linux, osvers=2.0.35, archname=i386-linux-thread
  uname='linux caliban.xs4all.nl 2.0.35 #2 sat jul 18 01​:37​:18 cest 1998 i686 unknown '
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define useperlio=undef d_sfio=undef
  Compiler​:
  cc='cc', optimize='-O2', gccversion=2.7.2.3
  cppflags='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'
  ccflags ='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'
  stdchar='char', d_stdstdio=define, usevfork=false
  intsize=4, longsize=4, ptrsize=4, doublesize=8
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt
  libc=, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Built under linux
  Compiled at Aug 9 1998 10​:17​:03
  @​INC​:
  /usr/lib/perl5/5.00502/i386-linux-thread
  /usr/lib/perl5/5.00502
  /usr/lib/perl5/site_perl/5.005/i386-linux-thread
  /usr/lib/perl5/site_perl/5.005
  .
[summer@​emu summer]$
This seems more probable​:
[summer@​emu summer]$ uname -a
Linux emu.os2.ami.com.au 2.2.12 #2 Sat Oct 2 21​:20​:39 WST 1999 i586 unknown
[summer@​emu summer]$

It also happens on RHL 6.1
[summer@​emu summer]$ perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 2) configuration​:
  Platform​:
  osname=linux, osvers=2.0.35, archname=i386-linux-thread
  uname='linux caliban.xs4all.nl 2.0.35 #2 sat jul 18 01​:37​:18 cest 1998 i686 unknown '
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define useperlio=undef d_sfio=undef
  Compiler​:
  cc='cc', optimize='-O2', gccversion=2.7.2.3
  cppflags='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'
  ccflags ='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'
  stdchar='char', d_stdstdio=define, usevfork=false
  intsize=4, longsize=4, ptrsize=4, doublesize=8
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt
  libc=, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Built under linux
  Compiled at Aug 9 1998 10​:17​:03
  @​INC​:
  /usr/lib/perl5/5.00502/i386-linux-thread
  /usr/lib/perl5/5.00502
  /usr/lib/perl5/site_perl/5.005/i386-linux-thread
  /usr/lib/perl5/site_perl/5.005
  .
[summer@​emu summer]$

--
Cheers
John Summerfield
http​://os2.ami.com.au/os2/ for OS/2 support.
Configuration, networking, combined IBM ftpsites index.

@p5pRT
Copy link
Author

p5pRT commented Dec 15, 1999

From @tamias

On Wed, Dec 15, 1999 at 09​:09​:45PM +0800, John Summerfield wrote​:

open(STDOUT, "| cat >foo.out") || die "Can't redirect stdout"; # JCS
open(STDERR, ">&STDOUT") || die "Can't dup stdout";

close(STDOUT);
close(STDERR);

This hangs (waits indefinitely), apparently at one or other of the closes
(lines 14/15). I presume cat's not getting EOF.

I see the same behavior here in 5.005_03. Stepping through with the
debugger, Perl hangs on close(STDOUT). Switching the order of the close()
statements, so that STDERR is closed before STDOUT, avoids the problem.

Ronald

@p5pRT
Copy link
Author

p5pRT commented Dec 16, 1999

From [Unknown Contact. See original ticket]

Ronald J Kimball <rjk@​linguist.dartmouth.edu> wrote

I see the same behavior here in 5.005_03. Stepping through with the
debugger, Perl hangs on close(STDOUT). Switching the order of the close()
statements, so that STDERR is closed before STDOUT, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a
pipe open. So it doesn't wait in close(STDERR);.

The prpoblem is a more general one, and will hit if the file descriptor
is copied in any way. For example, this code digs you into a similar
hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w
open FH, "| cat >temp.foo" or die "open​: $!\n";
my $pid = fork or sleep 1000000, exit;
print "Forked $pid\n";
close FH;
print "Closed\n";
__END__
Forked 5516

  ... then silence.

And of course, there's other twists if you close the filehandle in the
child from the fork, as it'll try to wait() for a PID which isn't it's
child.

Seems to me like a fundamental restriction in the way piped opens work,
which should be documented as "Don't do that, then."

Mike Guy

@p5pRT
Copy link
Author

p5pRT commented Dec 16, 1999

From [Unknown Contact. See original ticket]

Ronald J Kimball <rjk@​linguist.dartmouth.edu> wrote

I see the same behavior here in 5.005_03. Stepping through with the
debugger, Perl hangs on close(STDOUT). Switching the order of the close()
statements, so that STDERR is closed before STDOUT, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a
pipe open. So it doesn't wait in close(STDERR);.

The prpoblem is a more general one, and will hit if the file descriptor
is copied in any way. For example, this code digs you into a similar
hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w
open FH, "| cat >temp.foo" or die "open​: $!\n";
my $pid = fork or sleep 1000000, exit;
print "Forked $pid\n";
close FH;
print "Closed\n";
__END__
Forked 5516

... then silence.

For more than a million seconds? Eleven and a half days?
I change it to a more reasonable interval and it ran just fine.
[summer@​emu summer]$ testperl
Forked 9067
Closed
[summer@​emu summer]$

And of course, there's other twists if you close the filehandle in the
child from the fork, as it'll try to wait() for a PID which isn't it's
child.

Seems to me like a fundamental restriction in the way piped opens work,
which should be documented as "Don't do that, then."

That would break a lot of documentation (Camel book for one), and it seems
to me, is very unperl.

Lots of people are going to rely on existing printed material for years to
come, and there's a good chance this one will slip past the writers of new
documentation as well.

If there's way, pls find it.

--
Cheers
John Summerfield
http​://os2.ami.com.au/os2/ for OS/2 support.
Configuration, networking, combined IBM ftpsites index.

@p5pRT
Copy link
Author

p5pRT commented Jan 21, 2000

From @gsar

On Thu, 16 Dec 1999 18​:25​:15 GMT, "M.J.T. Guy" wrote​:

Ronald J Kimball <rjk@​linguist.dartmouth.edu> wrote

I see the same behavior here in 5.005_03. Stepping through with the
debugger, Perl hangs on close(STDOUT). Switching the order of the close()
statements, so that STDERR is closed before STDOUT, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a
pipe open. So it doesn't wait in close(STDERR);.

I don't think waiting in close(STDERR) is the problem. It's the close(STOUT)
that hangs when it waits. The subprocess won't get EOF until all the dup-ed
handles are closed too, so the parent waits indefinitely for the child to
quit, and the child waits indefinitely for the parent to close all the
handles, which it never will, because it is waiting for the child to quit.
Deadlock ensues.

The prpoblem is a more general one, and will hit if the file descriptor
is copied in any way. For example, this code digs you into a similar
hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w
open FH, "| cat >temp.foo" or die "open​: $!\n";
my $pid = fork or sleep 1000000, exit;
print "Forked $pid\n";
close FH;
print "Closed\n";
__END__
Forked 5516

... then silence.

As John noticed correctly, that doesn't hang; it merely sleeps a great long
while, because you asked for it. Here's a better example​:

  open(A, "| cat >foo.out") or die;
  open(B, ">&A") or die;
  print A "foo\n";

  close(A); # hangs...
  close(B); # ...unless you flip this line with the one above

The problem really originates in the old fclose/pclose duplicity. This
has the same issue​:

  #include <stdio.h>
  int
  main(int argc, char **argv)
  {
  FILE *f = popen("cat > foo.out", "w");
  if (f) {
  FILE *d = fdopen(dup(fileno(f)), "w");
  fprintf(f, "foo\n");
  pclose(f); /* hangs... */
  fclose(d); /* ...unless you flip this line with the one above */
  }
  }

Chalk up another reason to rewrite stdio.

Sarathy
gsar@​ActiveState.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant