Race condition in devel branch?

[p5pRT]

Migrated from rt.perl.org#1708 (status was 'resolved')

Searchable as RT1708$

[p5pRT]

From upf@de.uu.net

The problem can be reproduced with a miniperl build with gcc version
2.8.1.

The problem cannot be reproduced on Linux.

Any ideas?

Uli
-----------------------------------------------------------------
#!/users/opt/bin/perl -w
my @​fh = ();
$SIG{PIPE} = sub { die "whoops, $0 pipe broke" };

my $pid = open(PIPE_TO_KID, "|-");
if (not defined $pid) {
  die "Failed to fork​: $!";
} elsif (not $pid) {
  # child loop​: handle any data that is fed to the process via STDIN
  while (<>) {
  my ($slot, $text) = split /\t/;
  print "$$ got for $slot​: $text";
  if (not defined $fh[$slot]) { # output pipe does not exist yet
  $fh[$slot] = "$slot";
  my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
  print "$$​: using pipe $pipe\n";
  open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
  }
  print {$fh[$slot]} $text or die "Could not print to $slot​: $!";
  }
  for (@​fh) {
  close($_) or die "Could not close file handle​: $!" if $_;
  }
} else {
  # parent loop​: do some work, then send of any data to the child
  for (my $i=0; $i<2; $i++) {
  $| = 1; print "parent $$​: line $i\n";
  print PIPE_TO_KID "$i\tline $i\n" or die "Could not print​: $!";
  }
  close PIPE_TO_KID or die "$$​: Failed to close pipe​: $!";
}

Perl Info

Site configuration information for perl 5.00562:

Configured by techdb at Mon Oct 25 16:25:12 MET DST 1999.

Summary of my perl5 (revision 5.0 version 5 subversion 62) configuration:
  Platform:
    osname=solaris, osvers=2.6, archname=sun4-solaris
    uname='sunos billy20 5.6 generic_105181-16 sun4u sparc '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef useperlio=undef d_sfio=undef
    use64bits=undef usemultiplicity=undef
  Compiler:
    cc='gcc', optimize='-O', gccversion=egcs-2.91.66 19990314 (egcs-1.1.2 release)
    cppflags='-fno-strict-aliasing -I/usr/local/include'
    ccflags ='-fno-strict-aliasing -I/usr/local/include'
    stdchar='unsigned char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldl -lm -lc -lcrypt -lsec
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.00562:
    /tmp/test/install/lib/perl5/5.00562/sun4-solaris
    /tmp/test/install/lib/perl5/5.00562
    /tmp/test/install/lib/site_perl/5.00562/sun4-solaris
    /tmp/test/install/lib/site_perl
    .


Environment for perl 5.00562:
    HOME=/users/home/techdb
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/users/opt/lib:/usr/ccs/lib:/usr/openwin/lib:
    LOGDIR (unset)
    PATH=/users/opt/bin:/usr/local/bin:/usr/ccs/bin:/sbin:/usr/sbin:/usr/openwin/bin:/usr/bin:/usr/local/bin
    PERL_BADLANG (unset)
    SHELL=/users/opt/bin/bash

[p5pRT]

From [Unknown Contact. See original ticket]

Created by upf@de.uu.net

Did not receive that from the list ...

This is a bug report for perl from upf@​de.uu.net,
generated with the help of perlbug 1.27 running under perl 5.00562.
-----------------------------------------------------------------
The appended script writes two output files with a stable (5.005_03)
perl. When using a devel perl (57, 62) on a Solaris 2.6 system, it
often writes no files, sometimes one file but never two files.

The problem can be reproduced with a miniperl build with gcc version
2.8.1.

The problem cannot be reproduced on Linux.

Any ideas?

Uli
-----------------------------------------------------------------
#!/users/opt/bin/perl -w
my @​fh = ();
$SIG{PIPE} = sub { die "whoops, $0 pipe broke" };

my $pid = open(PIPE_TO_KID, "|-");
if (not defined $pid) {
  die "Failed to fork​: $!";
} elsif (not $pid) {
  # child loop​: handle any data that is fed to the process via STDIN
  while (<>) {
  my ($slot, $text) = split /\t/;
  print "$$ got for $slot​: $text";
  if (not defined $fh[$slot]) { # output pipe does not exist yet
  $fh[$slot] = "$slot";
  my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
  print "$$​: using pipe $pipe\n";
  open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
  }
  print {$fh[$slot]} $text or die "Could not print to $slot​: $!";
  }
  for (@​fh) {
  close($_) or die "Could not close file handle​: $!" if $_;
  }
} else {
  # parent loop​: do some work, then send of any data to the child
  for (my $i=0; $i<2; $i++) {
  $| = 1; print "parent $$​: line $i\n";
  print PIPE_TO_KID "$i\tline $i\n" or die "Could not print​: $!";
  }
  close PIPE_TO_KID or die "$$​: Failed to close pipe​: $!";
}

Perl Info

Site configuration information for perl 5.00562:

Configured by techdb at Mon Oct 25 16:25:12 MET DST 1999.

Summary of my perl5 (revision 5.0 version 5 subversion 62) configuration:
  Platform:
    osname=solaris, osvers=2.6, archname=sun4-solaris
    uname='sunos billy20 5.6 generic_105181-16 sun4u sparc '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef useperlio=undef d_sfio=undef
    use64bits=undef usemultiplicity=undef
  Compiler:
    cc='gcc', optimize='-O', gccversion=egcs-2.91.66 19990314 (egcs-1.1.2 release)
    cppflags='-fno-strict-aliasing -I/usr/local/include'
    ccflags ='-fno-strict-aliasing -I/usr/local/include'
    stdchar='unsigned char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldl -lm -lc -lcrypt -lsec
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.00562:
    /tmp/test/install/lib/perl5/5.00562/sun4-solaris
    /tmp/test/install/lib/perl5/5.00562
    /tmp/test/install/lib/site_perl/5.00562/sun4-solaris
    /tmp/test/install/lib/site_perl
    .


Environment for perl 5.00562:
    HOME=/users/home/techdb
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/users/opt/lib:/usr/ccs/lib:/usr/openwin/lib:
    LOGDIR (unset)
    PATH=/users/opt/bin:/usr/local/bin:/usr/ccs/bin:/sbin:/usr/sbin:/usr/openwin/bin:/usr/bin:/usr/local/bin
    PERL_BADLANG (unset)
    SHELL=/users/opt/bin/bash

[p5pRT]

From [Unknown Contact. See original ticket]

"U" == Ulrich Pfeifer <Ulrich.Pfeifer@​de.uu.net> writes​:

  U> Did not receive that from the list ...
  U> This is a bug report for perl from upf@​de.uu.net,
  U> generated with the help of perlbug 1.27 running under perl 5.00562.
  U> -----------------------------------------------------------------
  U> The appended script writes two output files with a stable (5.005_03)
  U> perl. When using a devel perl (57, 62) on a Solaris 2.6 system, it
  U> often writes no files, sometimes one file but never two files.

Two more data points​:
  The bug is not present in _54.
  It is still present in _63

  U> The problem can be reproduced with a miniperl build with gcc version
  U> 2.8.1.

  U> The problem cannot be reproduced on Linux.

  U> Any ideas?

  U> Uli

  #!/users/opt/bin/perl -w
  my @​fh = ();
  $SIG{PIPE} = sub { die "whoops, $0 pipe broke" };
 
  my $pid = open(PIPE_TO_KID, "|-");
  if (not defined $pid) {
  die "Failed to fork​: $!";
  } elsif (not $pid) {
  # child loop​: handle any data that is fed to the process via STDIN
  while (<>) {
  my ($slot, $text) = split /\t/;
  print "$$ got for $slot​: $text";
  if (not defined $fh[$slot]) { # output pipe does not exist yet
  $fh[$slot] = "$slot";
  my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
  print "$$​: using pipe $pipe\n";
  open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
  }
  print {$fh[$slot]} $text or die "Could not print to $slot​: $!";
  }
  for (@​fh) {
  close($_) or die "Could not close file handle​: $!" if $_;
  }
  } else {
  # parent loop​: do some work, then send of any data to the child
  for (my $i=0; $i<2; $i++) {
  $| = 1; print "parent $$​: line $i\n";
  print PIPE_TO_KID "$i\tline $i\n" or die "Could not print​: $!";
  }
  close PIPE_TO_KID or die "$$​: Failed to close pipe​: $!";
  }

Ulrich Pfeifer
--
I've also managed portable C++ projects. I would _much_ rather
herd cats. :-/ -- Kurt D. Starsinic

[p5pRT]

From [Unknown Contact. See original ticket]

The appended skript prints
,-----
| 1..2
| ok 1
`-----
with the offending versions. Removing the "open" line
produces the expected
,-----
| 1..2
| ok 1
| ok 2
`-----
(On Solaris 2.6)
Uli

---

#!perl

print "1..2\n"; $n=1;
my $pid = open(PIPE_TO_KID, "|-");
if (not defined $pid) {
  die "Failed to fork​: $!";
} elsif (not $pid) {
  # child loop​: handle any data that is fed to the process via STDIN
  my $slot;
  while (<>) {
  $fh[++$slot] = "$slot";
  my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
  print "ok $n\n"; $n++;
  open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
  }
} else {
  print PIPE_TO_KID "0\n1\n";
}

[p5pRT]

From [Unknown Contact. See original ticket]

Bug 19991025.003 is still there​:
,-----
| $ gtar xzf perl-5.5.670.tar.gz && rm perl-5.5.670.tar.gz
| $ cd perl-5.5.670
| $ ./Configure -Dcc=gcc -der && make miniperl
| $ ./miniperl -Ilib ../bug.pl
| 1..2
| ok 1
`-----
cat ../bug.pl
,-----
| #!perl
|
| print "1..2\n"; $n=1;
| my $pid = open(PIPE_TO_KID, "|-");
| if (not defined $pid) {
| die "Failed to fork​: $!";
| } elsif (not $pid) {
| # child loop​: handle any data that is fed to the process via STDIN
| my $slot;
| while (<>) {
| $fh[++$slot] = "$slot";
| my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
| print "ok $n\n"; $n++;
| open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
| }
| } else {
| print PIPE_TO_KID "0\n1\n";
| }
`-----

Ulrich Pfeifer
--
Summary of my perl5 (5.0 patchlevel 5 subversion 2) configuration​:
  Platform​:
  osname=solaris, osvers=2.6, archname=sun4-solaris
  uname='sunos shusai 5.6 generic_105181-03 sun4u sparc sunw,ultra-5_10 '
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef useperlio=undef d_sfio=undef
  Compiler​:
  cc='/users/opt/bin/gcc', optimize='undef', gccversion=
  cppflags='-I/usr/local/include'
  ccflags ='-I/usr/local/include'
  stdchar='unsigned char', d_stdstdio=define, usevfork=false
  intsize=4, longsize=4, ptrsize=4, doublesize=8
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries​:
  ld='/usr/local/bin/ld', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
  libs=-lsocket -lnsl -ldl -lm -lc -lcrypt
  libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags='-G -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Built under solaris
  Compiled at Nov 16 1998 20​:07​:35
  %ENV​:
  PERL5LIB="/home/upf/lib/perl5​:/home/upf/lib/perl5/site_perl"
  PERLLIB="/usr/local/pkg/perl5.005_03/lib/5.00503"
  @​INC​:
  /home/upf/lib/perl5
  /home/upf/lib/perl5/site_perl
  /users/opt/pkg/perl-5.005_02/lib/sun4-solaris
  /users/opt/pkg/perl-5.005_02/lib
  /users/opt/pkg/perl-5.005_02/lib/site_perl/5.005/sun4-solaris
  /users/opt/pkg/perl-5.005_02/lib/site_perl/5.005
  .

[p5pRT]

From @gsar

On 03 Mar 2000 09​:00​:01 +0100, Ulrich Pfeifer wrote​:

Bug 19991025.003 is still there​:
,-----
| $ gtar xzf perl-5.5.670.tar.gz && rm perl-5.5.670.tar.gz
| $ cd perl-5.5.670
| $ ./Configure -Dcc=gcc -der && make miniperl
| $ ./miniperl -Ilib ../bug.pl
| 1..2
| ok 1
`-----
cat ../bug.pl
,-----
| #!perl
|
| print "1..2\n"; $n=1;
| my $pid = open(PIPE_TO_KID, "|-");
| if (not defined $pid) {
| die "Failed to fork​: $!";
| } elsif (not $pid) {
| # child loop​: handle any data that is fed to the process via STDIN
| my $slot;
| while (<>) {
| $fh[++$slot] = "$slot";
| my $pipe = "|gzip -c >file.".sprintf("%04d", $slot).".gz";
| print "ok $n\n"; $n++;
| open($fh[$slot], $pipe) or die "Could not open pipe $pipe​: $!";
| }
| } else {
| print PIPE_TO_KID "0\n1\n";
| }
`-----

OK, I looked at this some, and it appears to be a bug in Solaris' stdio.
(Can't replicate the same problem with the test case below on Linux with
glibc, or even Windows.)

Here's a shorter version of the test (on Solaris)​:

  % cat > pipetest
  while (<STDIN>) {
  $n++; print "ok $n\n"; system "echo foo";
  }
  ^D
  % perl5.5.670 pipetest < pipetest
  ok 1
  foo
  ok 2
  foo
  ok 3
  foo
  % cat pipetest | perl5.5.670 pipetest
  ok 1
  foo
  %

So yes, if STDIN happens to be a pipe, you only get the first line (on
Solaris 2.6).

The bug seems to be triggered by the fflush(NULL) that we now do for system(),
fork() etc., to flush all the output buffers. stdin is not an output buffer,
but it still seems to lose its buffer anyways, but only if it happens to be a
pipe.

Here's a reduction of the failure into a short C test case​:

  #include <stdio.h>

  int
  main(int argc, char **argv)
  {
  char buf[1024];
  int i;
  char *bp = buf;
  while (1) {
  while ((i = getc(stdin)) != -1
  && (*bp++ = i) != '\n'
  && bp < &buf[1024])
  /* DO NOTHING */ ;
  *bp = '\0';
  fprintf(stdout, "%s", buf);
  fflush(NULL);
  if (i == -1)
  return 0;
  bp = buf;
  }
  }

And here is how it behaves on Solaris (note how it just prints one line
when fed from a pipe)​:

  % gcc -o ffnull ffnull.c
  % cat ffnull.c | ./ffnull
  #include <stdio.h>
  % ./ffnull < ffnull.c
  #include <stdio.h>

  int
  main(int argc, char **argv)
  {
  char buf[1024];
  int i;
  char *bp = buf;
  while (1) {
  while ((i = getc(stdin)) != -1
  && (*bp++ = i) != '\n'
  && bp < &buf[1024])
  /* DO NOTHING */ ;
  *bp = '\0';
  fprintf(stdout, "%s", buf);
  fflush(NULL);
  if (i == -1)
  return 0;
  bp = buf;
  }
  }
  %

Again, this test program behaves correctly under Linux with glibc-2.x and
on Windows.

If you're keen to avoid the bug on Solaris, you'd want to consider trading
one bug for another, and build Perl with​:

  ./Configure -des -Accflags='-DPERL_FLUSHALL_FOR_CHILD=NOOP'

(Note that that won't give you predictable flushing behavior on fork()
et al.)

Chalk up yet another reason to rewrite stdio for perl.

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

And here is how it behaves on Solaris (note how it just prints one line
when fed from a pipe)​:

% gcc -o ffnull ffnull.c
% cat ffnull.c | ./ffnull
#include <stdio.h>
...

Again, this test program behaves correctly under Linux with glibc-2.x and
on Windows.

Just FYI​: BSD tests ok, too.

Chalk up yet another reason to rewrite stdio for perl.

What ever happened to using sfio? I see notes in INSTALL about it.
Is there some licence or compat issue, or is retooling for sfio
part of what you meant?

--tom

[p5pRT]

From @gsar

On Sat, 04 Mar 2000 10​:30​:14 MST, Tom Christiansen wrote​:

Again, this test program behaves correctly under Linux with glibc-2.x and
on Windows.

Just FYI​: BSD tests ok, too.

Good.

Chalk up yet another reason to rewrite stdio for perl.

What ever happened to using sfio? I see notes in INSTALL about it.
Is there some licence or compat issue, or is retooling for sfio
part of what you meant?

Yup, using sfio is still within the realm of possibilities. Barring
license questions, the kicker is going to be whether writing it from
scratch is going to be harder than subverting sfio, which has a particular
mindset all its own.

I/O disciplines, for example. :-)

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

The bug seems to be triggered by the fflush(NULL) that we now do for system(),
fork() etc., to flush all the output buffers. stdin is not an output buffer,
but it still seems to lose its buffer anyways, but only if it happens to be a
pipe.

It doesn't matter whether you flush or don't flush, stdio is just
really not reliable enough for some kinds of work. And I don't
think you can fix this by retooling. Consider this code, which
uses read(2) and lseek(2)​:

  open(FH, ">mytmp") || die "mytmp​: $!";
  for(1..1000) { printf FH "%04d\n", $_ }
  close(FH) || die "mytmp​: $!";
  open(FH, "<mytmp") || die "mytmp​: $!";
  sysseek(FH, (-s FH)/2, 0) || die "sysseek​: $!";
 
  if ($pid = fork) { wait }
  else {
  die "cannot fork​: $!" unless defined $pid;
  sysread(FH, $_, 9) == 9 || die "sysread​: $!";
  print "Child $$ next 9 bytes are​: <$_>\n";
  exit;
  }
 
  sysread(FH, $_, 9) == 9 || die "sysread​: $!";
  print "Parent $$ next 9 bytes are​: <$_>\n";
  unlink("mytmp") || die "mytmp​: $!";

That of course works fine, producing

  Child 3258 next 9 bytes are​: <0501
  0502>
  Parent 3257 next 9 bytes are​: <
  0503
  050>

Now, consider what happens when you replace sys{read,seek} with the
{read,seek}, stdio's fread(3) and fseek(3). The shared seek pointer
on the inherited file descriptor is all screwed up because the child
did a read-ahead to fill its buffer.

This is the same issue as you see here, where

  % perl -le 'for(1..10_000){printf "%04d\n", $_}' | sed 10q
  0001
  0002
  0003
  0004
  0005
  0006
  0007
  0008
  0009
  0010

differs dramatically from​:

  % perl -le 'for(1..10_000){printf "%04d\n", $_}' | (sed 5q;sed 5q)
  0001
  0002
  0003
  0004
  0005

  3278
  3279
  3280
  3281

See what I mean? :-)

10q,

--tom

[p5pRT]

From [Unknown Contact. See original ticket]

Setting $|=1 on a filehandle not open for writing doesn't currently
emit a warning (although I'd swear I wrote a patch a long time ago
to do just that). Various intriguing possibilities come to mind. :-)

--tom

[p5pRT]

From [Unknown Contact. See original ticket]

"Sarathy" == Gurusamy Sarathy <gsar@​ActiveState.com> writes​:

  Sarathy> The bug seems to be triggered by the fflush(NULL) that we
  Sarathy> now do for system(), fork() etc., to flush all the output
  Sarathy> buffers. stdin is not an output buffer, but it still
  Sarathy> seems to lose its buffer anyways, but only if it happens
  Sarathy> to be a pipe.

Sarathy,

thanks for hunting that one down. I am sorry that I bothered you with
this. But it did brake some of our applications and I had some "Perl
is not reliable enough" discussions with colleagues.

Does somebody know if there is a patch from SUN for this? Alan?
Should we check for this Solaris bug in the regression tests?

Uli

[p5pRT]

From @AlanBurlison

Ulrich Pfeifer wrote​:

Does somebody know if there is a patch from SUN for this? Alan?
Should we check for this Solaris bug in the regression tests?

Bugid 4118926 was raised against this in 1988, and the same behaviour is
exhibited by Solaris 8.1. Opinion varied as to if it was a bug, or just
undefined behaviour. I'll append Sarathay's mail to the bug report and
see what sort of response I get.

--
Alan Burlison
Solaris Kernel Development, Sun Microsystems

[p5pRT]

From [Unknown Contact. See original ticket]

"Alan" == Alan Burlison <Alan.Burlison@​uk.sun.com> writes​:

  Alan> Ulrich Pfeifer wrote​:
  >> Does somebody know if there is a patch from SUN for this? Alan?
  >> Should we check for this Solaris bug in the regression tests?

  Alan> Bugid 4118926 was raised against this in 1988, and the same
  Alan> behaviour is exhibited by Solaris 8.1.

Thanks for looking this up Alan.
It's in there since 1988? Wow!

  Alan> Opinion varied as to if it was a bug, or just undefined
  Alan> behaviour.

What is fflush(NULL) useful for when eating data from input streams?

The fflush manual states​:

,-----
| If stream points to an output stream or an update stream in
| which the most recent operation was not input, fflush()
| causes any unwritten data for that stream to be written to
| the file, and the st_ctime and st_mtime fields of the under
| lying file are marked for update.
|
| If stream is a null pointer, fflush() performs this flushing
| action on all streams for which the behavior is defined
| above.
`-----

SUN should add before the final full stop​: "and performs some random
action on streams where the behavior is not defined"

  Alan> I'll append Sarathay's mail to the bug report and see what
  Alan> sort of response I get.

My I humbly suggest, "fixing" that somehow on the Perl side?
It breaks some common idiom like the famous​:

  @​ARGV = map { /\.gz$/ ? "gzip -cd $_|" : $_ } @​ARGV;

As in​:
,-----
| % ./miniperl -e '@​ARGV = "cat /etc/passwd|";
| while (<>) { print; system "echo" }'
| root​:x​:0​:1​:Super-User​:/​:/sbin/sh
| % wc /etc/passwd
| 17 30 672 /etc/passwd
`-----

Should we disable that on Solaris? Only if stdio is used? If flushing
of all output streams is a documented feature of Perl, we have to make
it work -- or we have to remove Solaris from the list of supported
systems.

Ulrich Pfeifer
--
perl -e 'print map chr($_&=127), unpack "C*", pack "w", \
"28031325277245338375695089309000712482357035071535370"'

[p5pRT]

From [Unknown Contact. See original ticket]

"Ulrich" == Ulrich Pfeifer <pfeifer@​wait.de> writes​:

"Alan" == Alan Burlison <Alan.Burlison@​uk.sun.com> writes​:

  Ulrich> Should we disable that on Solaris?

Probably Configure should set fflushNULL to 'undef' since it does not
work correctly. The fallback implementation in perl would do the
right thing then.

Ulrich Pfeifer
--
Assume competence. Train the ignorant. Fire the stupid.
-- Tom Christiansen

[p5pRT]

From @AlanBurlison

Ulrich Pfeifer wrote​:

Probably Configure should set fflushNULL to 'undef' since it does not
work correctly. The fallback implementation in perl would do the
right thing then.

That's my suggestion. If perl will fall back to 'doing the right
thing', that seems to be the sensible choice.

--
Alan Burlison
Solaris Kernel Development, Sun Microsystems

[p5pRT]

From @gsar

On Tue, 07 Mar 2000 12​:49​:45 GMT, Alan Burlison wrote​:

Ulrich Pfeifer wrote​:

Probably Configure should set fflushNULL to 'undef' since it does not
work correctly. The fallback implementation in perl would do the
right thing then.

That's my suggestion. If perl will fall back to 'doing the right
thing', that seems to be the sensible choice.

Setting fflushNULL to undef won't suffice to get the fallback
implementation that's in perl--you'll also need to set fflushall.

What's more, the fallback implementation won't cure the bug because
it will fflush() all the handles without regard to whether they're
(read) pipes or not.

One option is to disable flushing on fork/exec/system on Solaris, which
should be done with the addition to ccflags I mentioned. But this route
just trades one bug for another.

If there's any way to tell on Solaris whether an fd is the read end
of a pipe, we could work around the problem in my_fflush_all(). Is
there?

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

If there's any way to tell on Solaris whether an fd is the read end
of a pipe, we could work around the problem in my_fflush_all(). Is
there?

I seem to recall fstat()ing pipes and finding them marked both ways.
Ah, yes​: here it is.

% true | perl -MStat​::DumpFDs -e 'dump_dtable()' | cat
Using /home/tchrist/Stat-DumpFDs-0.01/blib
DUMPING up to 64 descriptors
fdno 0 is open and will be left open across execs.
fdno 0 is of type pipe
File is opened read-write.
flags on fdno 0 are​: [0x2]

fdno 1 is open and will be left open across execs.
fdno 1 is of type pipe
File is opened read-write.
flags on fdno 1 are​: [0x2]

fdno 2 is open and will be left open across execs.
fdno 2 is of type character special
File is opened read-write.
flags on fdno 2 are​: [0x2]

--tom

[p5pRT]

From @AlanBurlison

Gurusamy Sarathy wrote​:

If there's any way to tell on Solaris whether an fd is the read end
of a pipe, we could work around the problem in my_fflush_all(). Is
there?

Sorta. In 2.7 onwards he appropriate functions are __freading(FILE*)
and __fwriting(FILE*), but unfortunately these aren't in any earlier
versions of the OS - you would have to grot around in the innards of the
stdio structures. The appropriate manpage can be found at
http​://docs.sun.com​:80/ab2/coll.40.6/REFMAN3A/@​Ab2PageView/75495. Note
also that these are uncommited (i.e. changeable) interfaces, and aren't
portable.

--
Alan Burlison
Solaris Kernel Development, Sun Microsystems

[p5pRT]

From [Unknown Contact. See original ticket]

"Sarathy" == Gurusamy Sarathy <gsar@​ActiveState.com> writes​:

  Sarathy> One option is to disable flushing on fork/exec/system on
  Sarathy> Solaris, which should be done with the addition to
  Sarathy> ccflags I mentioned. But this route just trades one bug
  Sarathy> for another.

Sorry, did not look closely enough.

Setting fflushNULL correctly will have the same effect than defining
PERL_FLUSHALL_FOR_CHILD except for the installations using SFIO. I'd
prefer the test instead of manually defining PERL_FLUSHALL_FOR_CHILD
if we cannot fix it.

Ulrich Pfeifer
--
My program has just dumped Stova Core!
-- "If Klingons Developed Software" by Jarkko Hietaniemi

[p5pRT]

From @gsar

On 08 Mar 2000 00​:08​:11 +0100, Ulrich Pfeifer wrote​:

Setting fflushNULL correctly will have the same effect than defining
PERL_FLUSHALL_FOR_CHILD except for the installations using SFIO. I'd
prefer the test instead of manually defining PERL_FLUSHALL_FOR_CHILD
if we cannot fix it.

Configure should run the simple test case I showed (after it runs the
existing tests for fflush(NULL)) to determine if fflushNULL should in
fact be set.

Can you write a patch for that?

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

"Sarathy" == Gurusamy Sarathy <gsar@​ActiveState.com> writes​:

  Sarathy> Configure should run the simple test case I showed (after
  Sarathy> it runs the existing tests for fflush(NULL)) to determine
  Sarathy> if fflushNULL should in fact be set.

  Sarathy> Can you write a patch for that?

Hope this is what you had in mind. It does the right thing on Linux

,-----
| Checking how to flush all pending stdio output...
| Your fflush(NULL) works okay for output streams.

| Let's see if it clobbers input pipes...
| fflush(NULL) seems not to clobber input streams. We will use it then.
`-----

,-----
| % grep fflush config.sh
| # Source directory : /usr/src/perl/perl-5.6.0-RC1-fflush
| fflushNULL='define'
| fflushall='undef'
`-----

and Solaris 2.6​:

,-----
| Checking how to flush all pending stdio output...
| Your fflush(NULL) works okay for output streams.

| Let's see if it clobbers input pipes...
| Ouch, fflush(NULL) clobbers input pipes! We will not use it.
| (Now testing the other method--but note that also this may fail.)
| Segmentation Fault - core dumped
| Sigh. Flushing explicitly all the stdio streams doesn't work.
`-----

,-----
| % grep fflush config.sh
| fflushNULL='undef'
| fflushall='undef'
`-----

I did not have the current metaconfig units. So the patch must be
retrofitted :-(

Uli

diff -c 'perl-5.6.0-RC1/Configure' 'perl-5.6.0-RC1-fflush/Configure'
Index​: ./Configure
Prereq​: 3.0.1.9
*** ./Configure Mon Mar 6 05​:56​:00 2000
--- ./Configure Sat Mar 11 18​:20​:19 2000
***************
*** 12553,12561 ****
  $rm -f core try.core core.try.*
  case "$fflushNULL" in
  x) $cat >&4 <<EOM
! Your fflush(NULL) works okay.
  EOM
! fflushNULL="$define"
  ;;
  '') $cat >&4 <<EOM
  Your fflush(NULL) isn't working (contrary to ANSI C).
--- 12553,12605 ----
  $rm -f core try.core core.try.*
  case "$fflushNULL" in
  x) $cat >&4 <<EOM
! Your fflush(NULL) works okay for output streams.
! Let's see if it clobbers input pipes...
  EOM
! # All versions of Solaris appear to have a stdio bug that improperly
! # flushes the input end of pipes. So we avoid the autoflush on
! # fork/system/exec support for now. :-(
! $cat >tryp.c <<EOCP
! #include <stdio.h>
!
! int
! main(int argc, char **argv)
! {
! char buf[1024];
! int i;
! char *bp = buf;
! while (1) {
! while ((i = getc(stdin)) != -1
! && (*bp++ = i) != '\n'
! && bp < &buf[1024])
! /* DO NOTHING */ ;
! *bp = '\0';
! fprintf(stdout, "%s", buf);
! fflush(NULL);
! if (i == -1)
! return 0;
! bp = buf;
! }
! }
! EOCP
! fflushNULL="$define"
! set tryp
! if eval $compile; then
! $rm -f tryp.out
! $cat tryp.c | ./tryp$exe_ext 2>/dev/null > tryp.out
! if cmp tryp.c tryp.out >/dev/null 2>&1; then
! $cat >&4 <<EOM
! fflush(NULL) seems not to clobber input streams. We will use it then.
! EOM
! fflushNULL="$define"
! else
! $cat >&4 <<EOM
! Ouch, fflush(NULL) clobbers input pipes! We will not use it.
! EOM
! fflushNULL="$undef"
! fi
! fi
! $rm -f core tryp.core core.tryp.*
  ;;
  '') $cat >&4 <<EOM
  Your fflush(NULL) isn't working (contrary to ANSI C).
diff -c 'perl-5.6.0-RC1/hints/solaris_2.sh' 'perl-5.6.0-RC1-fflush/hints/solaris_2.sh'
Index​: ./hints/solaris_2.sh
*** ./hints/solaris_2.sh Thu Mar 9 13​:42​:35 2000
--- ./hints/solaris_2.sh Sat Mar 11 17​:02​:02 2000
***************
*** 479,507 ****
  ;;
  esac
 
- case "$fflushNULL" in
- "$define"|true|[yY]*)
- # allow them to force it the other way
- ;;
- *)
- # All versions of Solaris appear to have a stdio bug that improperly
- # flushes the input end of pipes. So we avoid the autoflush on
- # fork/system/exec support for now. :-( See the test case in​:
- # http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-03/msg00373.html
- # XXX this needs a Configure test so that more such platforms can be
- # caught automatically.
- fflushNULL=undef
- fflushall=undef
- cat >&4 <<EOM
-
- *** Solaris has a bug that affects input pipes, and so I have disabled
- *** support for automatic flushing on fork/system/exec. If you want
- *** automatic flushing anyway, rerun Configure again with -DfflushNULL.
-
- EOM
- ;;
- esac
-
  rm -f try.c try.o try
  # keep that leading tab
  ccisworkshop=''
--- 479,484 ----

[p5pRT]

From @jhi

Patch applied, thanks.

--
$jhi++; # http​://www.iki.fi/jhi/
  # There is this special biologist word we use for 'stable'.
  # It is 'dead'. -- Jack Cohen

[p5pRT]

From @gsar

On 11 Mar 2000 18​:47​:25 +0100, Ulrich Pfeifer wrote​:

and Solaris 2.6​:

,-----
| Checking how to flush all pending stdio output...
| Your fflush(NULL) works okay for output streams.

| Let's see if it clobbers input pipes...
| Ouch, fflush(NULL) clobbers input pipes! We will not use it.
| (Now testing the other method--but note that also this may fail.)
| Segmentation Fault - core dumped
| Sigh. Flushing explicitly all the stdio streams doesn't work.
`-----

Good thing we did this in Configure--I just found that HP-UX 11.00 suffers
from the same braindamage. :-(

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

Gurusamy Sarathy <gsar@​ActiveState.com> wrote

Good thing we did this in Configure--I just found that HP-UX 11.00 suffers
from the same braindamage. :-(

What is Perl's backup strategy if this fails? Do no flushing?

This quote from "perldoc -f fork" presumably needs appropriate weasel
words adding. What should they be?

  All files opened for output are flushed before
  forking the child process.

Mike Guy

[p5pRT]

From @doughera88

On Sat, 11 Mar 2000, Gurusamy Sarathy wrote​:

On 11 Mar 2000 18​:47​:25 +0100, Ulrich Pfeifer wrote​:

and Solaris 2.6​:

,-----
| Checking how to flush all pending stdio output...
| Your fflush(NULL) works okay for output streams.

| Let's see if it clobbers input pipes...
| Ouch, fflush(NULL) clobbers input pipes! We will not use it.
| (Now testing the other method--but note that also this may fail.)
| Segmentation Fault - core dumped
| Sigh. Flushing explicitly all the stdio streams doesn't work.
`-----

Good thing we did this in Configure--I just found that HP-UX 11.00 suffers
from the same braindamage. :-(

Beware, though, that the fall-back of explicitly flushing all the stdio
streams is quite brute force and makes no attempt whatsoever to
distinguish among different types of input or output. It does not
distinguish between input and output. It does not distinguish between
pipes and files. It's pretty crude. I'd be rather surprised if there
were ever a platform where fflush(NULL) clobbered input pipes but util.c's
simple PerlIO_my_fflush_all() function didn't.

Perhaps someone with a few minutes to poke around stdio flags might
be able to improve on the very crude test which currently is just

  if (STDIO_STREAM_ARRAY[i]._file >= 0 &&
  STDIO_STREAM_ARRAY[i]._file < open_max &&
  STDIO_STREAM_ARRAY[i]._flag)
  PerlIO_flush(&STDIO_STREAM_ARRAY[i]);

Poking around the sundry bits of _flag might be useful, if not
particularly portable. (As long as the Configure test stays in
sync with the util.c test, this won't introduce any new core dumps.)

--
  Andy Dougherty doughera@​lafayette.edu
  Dept. of Physics
  Lafayette College, Easton PA 18042

[p5pRT]

From @gsar

On Sun, 12 Mar 2000 14​:53​:16 GMT, "M.J.T. Guy" wrote​:

Gurusamy Sarathy <gsar@​ActiveState.com> wrote

Good thing we did this in Configure--I just found that HP-UX 11.00 suffers
from the same braindamage. :-(

What is Perl's backup strategy if this fails? Do no flushing?

Yes. (At least until we get rid of stdio.)

This quote from "perldoc -f fork" presumably needs appropriate weasel
words adding. What should they be?

        All files opened for output are flushed before
        forking the child process\.

Something to the effect of : "... if the platform's I/O implementation
cooperates."

Sarathy
gsar@​ActiveState.com

[p5pRT]

From @gsar

On Sun, 12 Mar 2000 10​:18​:27 EST, Andy Dougherty wrote​:

Beware, though, that the fall-back of explicitly flushing all the stdio
streams is quite brute force and makes no attempt whatsoever to
distinguish among different types of input or output.

We now test for fflush(stdin) as a precondition for attempting to determine
if we should try fflushall. As it happens, both Solaris and HP-UX fail
fflush(stdin), so they never even run the explicit flushing test.

Sarathy
gsar@​ActiveState.com

[p5pRT]

From [Unknown Contact. See original ticket]

Gurusamy Sarathy <gsar@​ActiveState.com> wrote

Something to the effect of : "... if the platform's I/O implementation
cooperates."

How's this?

The list of platforms in perlport needs checking. Does the problem
apply to all versions of Solaris 2? And to all versions of HP-UX?
What other platforms are affected.

Mike Guy

Inline Patch

--- ./pod/perlfunc.pod.orig	Thu Mar  9 17:40:40 2000
+++ ./pod/perlfunc.pod	Mon Mar 13 09:27:50 2000
@@ -1651,6 +1651,10 @@
 dominant paradigm for multitasking over the last few decades.
 
 All files opened for output are flushed before forking the child process.
+But be warned that this does not happen on some platforms.   Nor did
+it happen in Perl versions prior to 5.6.0.    So you may need to
+set C<$|> ($AUTOFLUSH in English) or call the C<autoflush()>
+method of C<IO::Handle> to avoid duplicate output.
 
 If you C<fork> without ever waiting on your children, you will
 accumulate zombies.  On some systems, you can avoid this by setting
--- ./pod/perlport.pod.orig	Fri Mar  3 18:55:45 2000
+++ ./pod/perlport.pod	Mon Mar 13 09:30:38 2000
@@ -1301,6 +1301,8 @@
 
 Not implemented. (S<Mac OS>, Win32, AmigaOS, S<RISC OS>, VOS, VM/ESA)
 
+Does not flush output filehandles on some platforms.  (Solaris, HP-UX)
+
 =item getlogin
 
 Not implemented. (S<Mac OS>, S<RISC OS>)

End of patch