Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bleadperl breaks OALDERS/LWP-Protocol-https-6.07.tar.gz on Windows #16418

Closed
p5pRT opened this issue Feb 13, 2018 · 26 comments
Closed

Bleadperl breaks OALDERS/LWP-Protocol-https-6.07.tar.gz on Windows #16418

p5pRT opened this issue Feb 13, 2018 · 26 comments
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s) distro-mswin32 type-core
Milestone

Comments

@p5pRT
Copy link

p5pRT commented Feb 13, 2018

Migrated from rt.perl.org#132863 (status was 'resolved')

Searchable as RT132863$

@p5pRT
Copy link
Author

p5pRT commented Feb 13, 2018

From @chorny

Created by @chorny

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: libwww-perl/LWP-Protocol-https#52

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.27.8:

Configured by strawberry-perl at Tue Feb 13 01:56:34 2018.

Summary of my perl5 (revision 5 version 27 subversion 8) configuration:

  Platform:
    osname=MSWin32
    osvers=5.1.2600
    archname=MSWin32-x86-multi-thread-64int
    uname='Win32 strawberry-perl 5.27.8.1-beta1 #1 Tue Feb 13 01:53:53
2018 i386'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -D__USE_MINGW_ANSI_STDIO
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=4
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=12
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++.exe'
    ldflags ='-s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"'
    libpth=H:\perl\278\c\lib H:\perl\278\c\i686-w64-mingw32\lib
H:\perl\278\c\lib\gcc\i686-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl527.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"'



@INC for perl 5.27.8:
    H:/perl/278/perl/site/lib
    H:/perl/278/perl/vendor/lib
    H:/perl/278/perl/lib


Environment for perl 5.27.8:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=H:\perl\278\perl\site\bin;H:\perl\278\perl\bin;H:\perl\278\c\bin;C:\Program
Files\Far\;C:\Prg\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
Files\ATI Technologies\ATI.ACE\Core-Static;C:\strawberry514\c\bin;C:\strawberry514\perl\site\bin;C:\strawberry514\perl\bin;C:\Program
Files\TortoiseHg\;C:\Program Files\MySQL\MySQL Server
5.5\bin;c:\util\;C:\Prg\Subversion\bin;C:\Program
Files\TortoiseSVN\bin;D:\vagrant\vagrant\bin;C:\Prg\TortoiseGit\bin;C:\Prg\Git\cmd;C:\Program
Files\Skype\Phone\;
    PERL_BADLANG (unset)
    SHELL (unset)


-- 
Alexandr Ciornii, http://chorny.net

@p5pRT
Copy link
Author

p5pRT commented Feb 14, 2018

From @jkeenan

On Tue, 13 Feb 2018 11​:43​:20 GMT, chorny wrote​:

This is a bug report for perl from alexchorny@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.27.8.

-----------------------------------------------------------------
[Please describe your issue here]

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: https://github.com/libwww-perl/LWP-Protocol-
https/issues/52

I see that the test file which is hanging has code in it like this​:

#####
defined( my $pid = fork()) or die "fork failed​: $!";

# child process runs _server and exits
if ( ! $pid ) {
  @​childs = ();
  exit( _server());
}
#####

... and as well​:

#####
$ grep -n goto t/https_proxy.t
205​: my $cl = $server->accept or goto ACCEPT;
215​: goto ACCEPT;
230​: goto ACCEPT; # wait for next connection
285​: goto SSL_UPGRADE;
306​: goto REQUEST if $keep_alive;
#####

'goto' is cited in commit messages for the following commits in the last 3 months​:

#####
436908e
b537774
e7afb05
a01f464
e57923a
da4e040
e57923a
6d90e98
4d7e83b
4bfb553
#####

Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

List​: Do people on other versions of Windows also get this hang?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Feb 14, 2018

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 7, 2018

From @chorny

2018-02-14 16​:56 GMT+02​:00 James E Keenan via RT <perlbug-followup@​perl.org>​:

On Tue, 13 Feb 2018 11​:43​:20 GMT, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.
Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings". Commit 503bc07
does not have such problems.
Other affected distributions are Test-WWW-Mechanize-Catalyst and
Test-HTTP-Server-Simple and possibly others.

When installing IO​::Socket​::SSL, you should install
MIKEM/Net-SSLeay-1.82.tar.gz first and skip it's test
t/local/33_x509_create_cert.t.

--
Alexandr Ciornii, http​://chorny.net

@p5pRT
Copy link
Author

p5pRT commented Apr 21, 2018

From @chorny

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

--
Alexandr Ciornii, http​://chorny.net

@p5pRT
Copy link
Author

p5pRT commented Apr 21, 2018

From [Unknown Contact. See original ticket]

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

--
Alexandr Ciornii, http​://chorny.net

@p5pRT
Copy link
Author

p5pRT commented Apr 22, 2018

From @jkeenan

On Sat, 21 Apr 2018 22​:51​:22 GMT, chorny wrote​:

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1,
5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz
(https://rt.cpan.org/Ticket/Display.html?id=125169)

It appears that we neglected to add this RT to the list of perl-5.28.0 blockers (RT 127689). I have done so now.

These failures appear to be Windows-specific. Just now I used cpanm to successfully install every module listed in this RT on both Linux and FreeBSD-11.1.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 23, 2018

From @iabyn

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

  END { kill 9,@​childs if @​childs };

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

Do you have any insight?

--
Dave's first rule of Opera​:
If something needs saying, say it​: don't warble it.

@p5pRT
Copy link
Author

p5pRT commented Apr 30, 2018

From @iabyn

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@&#8203;childs if @&#8203;childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

  LWP-Protocol-https-6.0
  KAORU/HTTP-Async-0.33
  IO​::Socket​::SSL

And ideally confirm my diagnosis that on fork() on win32, END()
blocks were being called twice in the child (and presumably once in the
parent), but only once after my patch. If someone could write a test for
this, that would be even better,

Here's the commit message​:

  win323 fork()​: honour PERL_EXIT_DESTRUCT_END
 
  RT #132863
 
  The PERL_EXIT_DESTRUCT_END flag in PL_exit_flags is designed to defer
  the calling of END blocks in perl_run() to being called from
  perl_destruct() instead. On UNIX-like builds, perlmain.c sets this flag.
  So main() looks like, in outline​:
 
  PL_exit_flags |= PERL_EXIT_DESTRUCT_END;
  if (!perl_parse(my_perl, xs_init, argc, argv, (char **)NULL))
  perl_run(my_perl);
  exitstatus = perl_destruct(my_perl);
 
  which means that it doesn't matter whether perl_parse() finishes
  normally or prematurely (e.g. via BEGIN { exit(1) } or BEGIN { die });
  in all cases, due to PERL_EXIT_DESTRUCT_END being set, the END blocks
  will always be called (from perl_destruct()).
 
  Commit v5.27.7-9-g8e920bd341 added PERL_EXIT_DESTRUCT_END to the
  equivalent of main() on other platforms such as win32; this
  means that the new tests in t/op/blocks.t pass on win32 too, and all
  platforms have the same behaviour for e.g.
 
  END { print "end\n"; } BEGIN { exit 1; } # prints "end"
 
  However, that commit was causing some tests in some CPAN distributions to
  hang. These were using the win32 fork() emulation.
 
  PerlProcFork() on win32 clones an interpreter, starts a new thread,
  and makes the new thread call win32_start_child(), which
  does the rough equivalent of C<perl_run(); perl_destruct();>,
  except that it rolls it's own perl_run() equivalent which
  *doesn't* honour the PERL_EXIT_DESTRUCT_END flag. So by setting
  PERL_EXIT_DESTRUCT_END, END blocks in fork()ed processes were getting
  executed twice​: once by win32_start_child() emulating perl_run(),
  and once by perl_destruct() called from win32_start_child().
 
  This commit makes win32_start_child() honour PERL_EXIT_DESTRUCT_END.

--
It's not that I'm afraid to die, I just don't want to be there when it
happens.
  -- Woody Allen

@p5pRT
Copy link
Author

p5pRT commented May 3, 2018

From @steve-m-hay

On 30 April 2018 at 22​:15, Dave Mitchell <davem@​iabyn.com> wrote​:

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@&#8203;childs if @&#8203;childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO&#8203;::Socket&#8203;::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs
before doing anything. With your branch it now passes all 8 tests but
then hangs instead of exiting. So it's an improvement of sorts, but
something is still amiss somewhere. However, there's no improvement
with t/session_ticket.t. With either build I get the following
failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t
1..6
# listen at 127.0.0.1​:51973
# listen at 127.0.0.1​:51974
access to server[0]
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 1 - no initial session -> no reuse
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 2 - reuse with the next session and secret[0]
access to server[1]
using current ticket secret
server[1] reused=1
rotate secrets
# connect to 1​: success reuse=1
ok 3 - reuse even though server changed, since they share ticket secret
access to server[1]
using non-current ticket secret
creating new ticket key2
server[1] reused=0
rotate secrets
# connect to 1​: success reuse=0
ok 4 - reports non-reuse since server1 changed secret to secret[1]
access to server[0]
using non-current ticket secret
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 6 - reuse again since got ticket with secret[0] in last step
not ok 1 - select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
#
# Failed test 'select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
# '
# at ./t/testlib.pl line 39.

I haven't tried an earlier
(pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that
this used to work. I will try to do that in the next day or two.

@p5pRT
Copy link
Author

p5pRT commented May 4, 2018

From @steve-m-hay

On 3 May 2018 at 18​:20, Steve Hay <steve.m.hay@​googlemail.com> wrote​:

On 30 April 2018 at 22​:15, Dave Mitchell <davem@​iabyn.com> wrote​:

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@&#8203;childs if @&#8203;childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO&#8203;::Socket&#8203;::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs
before doing anything. With your branch it now passes all 8 tests but
then hangs instead of exiting. So it's an improvement of sorts, but
something is still amiss somewhere. However, there's no improvement
with t/session_ticket.t. With either build I get the following
failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t
1..6
# listen at 127.0.0.1​:51973
# listen at 127.0.0.1​:51974
access to server[0]
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 1 - no initial session -> no reuse
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 2 - reuse with the next session and secret[0]
access to server[1]
using current ticket secret
server[1] reused=1
rotate secrets
# connect to 1​: success reuse=1
ok 3 - reuse even though server changed, since they share ticket secret
access to server[1]
using non-current ticket secret
creating new ticket key2
server[1] reused=0
rotate secrets
# connect to 1​: success reuse=0
ok 4 - reports non-reuse since server1 changed secret to secret[1]
access to server[0]
using non-current ticket secret
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 6 - reuse again since got ticket with secret[0] in last step
not ok 1 - select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
#
# Failed test 'select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
# '
# at ./t/testlib.pl line 39.

I haven't tried an earlier
(pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that
this used to work. I will try to do that in the next day or two.

I've now tried a build of the change prior to
8e920bd (namely,
503bc07) and unfortunately both
t/mitm.t and t/session_ticket.t pass all tests (and don't hang).
That's on the same machine with the same compiler and build
configuration.

So the problem here isn't yet fixed.

@p5pRT
Copy link
Author

p5pRT commented May 4, 2018

From @iabyn

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

  $| = 1;
  my $pid = fork();
  die "fork​: $!\n" unless defined $pid;
  my $who = $pid ? "parent" : "child";
  END { print "END​: $who\n" }

--
Overhead, without any fuss, the stars were going out.
  -- Arthur C Clarke

@p5pRT
Copy link
Author

p5pRT commented May 4, 2018

From @steve-m-hay

On 4 May 2018 at 20​:39, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

@p5pRT
Copy link
Author

p5pRT commented May 8, 2018

From @iabyn

On Fri, May 04, 2018 at 09​:36​:17PM +0100, Steve Hay via perl5-porters wrote​:

On 4 May 2018 at 20​:39, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

Hmmm.

Could you show the output from the attached script for each of the perls
described above, thanks?

--
I before E. Except when it isn't.

@p5pRT
Copy link
Author

p5pRT commented May 8, 2018

From @iabyn

#!/usr/bin/perl

use warnings;
use strict;

my ($pid, @​pid);
my $type = 'parent';

END {
  print STDERR "END​: $type​: killing (@​pid)\n";
  kill 9,@​pid;
  print STDERR "END​: $type​: done kill (@​pid)\n";
}

$pid = fork;
die $! unless defined $pid;
if (!$pid) {
  $type = 'child';
  while (1) {
  print STDERR "child​: sleep(1)\n";
  sleep(1);
  }
  print STDERR "child​: exit(1)\n";
  exit(1);
}

push @​pid, $pid;
print STDERR "parent​: sleep(4)\n";
sleep(4);
print STDERR "parent​: finishing\n";

@p5pRT
Copy link
Author

p5pRT commented May 8, 2018

From @steve-m-hay

On 8 May 2018 at 14​:27, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 09​:36​:17PM +0100, Steve Hay via perl5-porters wrote​:

On 4 May 2018 at 20​:39, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

Hmmm.

Could you show the output from the attached script for each of the perls
described above, thanks?

Pre-8e920bd341 outputs this​:

parent​: sleep(4)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
parent​: finishing
END​: parent​: killing (-12584)
END​: parent​: done kill (-12584)

Blead and your branch both output this​:

parent​: sleep(4)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
parent​: finishing
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
child​: sleep(1)
[...] Ctrl+C to kill it
Terminating on signal SIGINT(2)

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @iabyn

On Tue, May 08, 2018 at 05​:57​:10PM +0100, Steve Hay via perl5-porters wrote​:

Pre-8e920bd341 outputs this​:
...
END​: parent​: killing (-12584)

Blead and your branch both output this​:
[...] Ctrl+C to kill it

Thanks for that, I finally know what's going on.

TL;DR​: a can of worms has been opened, and for 5.28 we should close
the can by reverting for now the 8e920bd PERL_EXIT_DESTRUCT_END
change (plus the tests), then re-address this issue post 5.28.

Long form​:

As I mentioned earlier in this thread, the main() function on UNIXYy
builds looks like

  PL_exit_flags |= PERL_EXIT_DESTRUCT_END;
  if (!perl_parse(my_perl...)
  perl_run(my_perl);
  exitstatus = perl_destruct(my_perl);

Setting the PERL_EXIT_DESTRUCT_END flag moves the calling of END blocks
from being handled at the end of perl_run() to being handled at the
beginning of perl_destruct(). This means that a premature exit
from perl_parse(), such as BEGIN { exit(1) }, doesn't skip calling END
blocks.

Zefram's 8e920bd commit merely set the PERL_EXIT_DESTRUCT_END flag on
other platforms such as win32, so that they too would call END blocks
even after BEGIN { exit(1) }.

However, calling END blocks isn't *quite* the first thing in
perl_destruct(); before that it does

  /* wait for all pseudo-forked children to finish */
  PERL_WAIT_FOR_CHILDREN;

so by setting PERL_EXIT_DESTRUCT_END, we've moved the calling of
END blocks to *after* the reaping of pseudo-forked children.

It so happens that of the failing distributions and tests that I looked
at, they shared a common idiom​: fork() a child to start a test server
which can be used to run tests against, then at the end, kill off any such
children with

  END { kill 9, @​child_pids }

Now on win32, the END block won't get called until after the children have
finished, but the children won't finish until the have been killed by the
parent calling the END block. Hence deadlock.

The fix may well be to move PERL_WAIT_FOR_CHILDREN to after
the call_list(PL_endav) in perl_destruct(), but I think messing about like
that in late code freeze is asking for trouble,

There are also wider issues. For example, one of the short bits of code
Steve ran for me earlier showed that the win32 fork() emulation wasn't
calling END() blocks in the child.

We need to look at all the cases and decide whether and when END blocks
should be called, and if so fix them on platforms where they don't match
expectations , e.g.

  BEGIN { die }
  BEGIN { exit 0 }
  BEGIN { exit 1 }
  die
  exit 0
  exit 1
  fork; then in child​: eval 'BEGIN { die }'
  fork; then in child​: eval 'BEGIN { exit 0 }'
  fork; then in child​: eval 'BEGIN { exit 1 }'
  fork; then in child​: die
  fork; then in child​: exit 0
  fork; then in child​: exit 1
  fork; then from parent, kill child
  fork; then from parent, kill 9, child

--
Hofstadter's Law​: It always takes longer than you expect, even when you
take into account Hofstadter's Law.

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @iabyn

On Wed, May 09, 2018 at 09​:48​:27AM +0100, Dave Mitchell wrote​:

TL;DR​: a can of worms has been opened, and for 5.28 we should close
the can by reverting for now the 8e920bd PERL_EXIT_DESTRUCT_END
change (plus the tests), then re-address this issue post 5.28.

One final favour, Steve​:

Could you run blead with 8e920bd
reverted, and and show me which tests t/op/blocks.t (or elsewhere) fail?
Its' hard to determine which tests to skip without actually running it on a
win32 platform.

Thanks.

--
This email is confidential, and now that you have read it you are legally
obliged to shoot yourself. Or shoot a lawyer, if you prefer. If you have
received this email in error, place it in its original wrapping and return
for a full refund. By opening this email, you accept that Elvis lives.

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @steve-m-hay

On 9 May 2018 at 11​:04, Dave Mitchell <davem@​iabyn.com> wrote​:

On Wed, May 09, 2018 at 09​:48​:27AM +0100, Dave Mitchell wrote​:

TL;DR​: a can of worms has been opened, and for 5.28 we should close
the can by reverting for now the 8e920bd PERL_EXIT_DESTRUCT_END
change (plus the tests), then re-address this issue post 5.28.

One final favour, Steve​:

Could you run blead with 8e920bd
reverted, and and show me which tests t/op/blocks.t (or elsewhere) fail?
Its' hard to determine which tests to skip without actually running it on a
win32 platform.

Thanks.

Current blead with 8e920bd reverted
passes all tests except t/op/blocks.t tests 10, 11, 13, 14, 16 and 17.
Verbose output from this test is attached.

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @steve-m-hay

# Failed test 10 - BEGIN{exit 1} should exit at ./test.pl line 1062
# got "begin\nunitcheck\ncheck"
# expected "begin\nunitcheck\ncheck\nend"
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; BEGIN { exit 1; }
# STATUS​: 256
# Failed test 11 - BEGIN{die} should exit at ./test.pl line 1074
# got 'begin\nDied at - line 1.\nBEGIN failed--compilation aborted at - line 1.\nunitcheck\ncheck'
# expected /(?^​:\Abegin\nDied[^\n]*\.\nBEGIN failed[^\n]*\.\nunitcheck\ncheck\nend\z)/
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; BEGIN { die; }
# STATUS​: 65280
# Failed test 13 - UNITCHECK{exit 1} should exit at ./test.pl line 1062
# got "begin\nunitcheck\ncheck"
# expected "begin\nunitcheck\ncheck\nend"
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; UNITCHECK { exit 1; }
# STATUS​: 256
# Failed test 14 - UNITCHECK{die} should exit at ./test.pl line 1074
# got 'begin\nDied at - line 1.\nUNITCHECK failed--call queue aborted.\nunitcheck\ncheck'
# expected /(?^​:\Abegin\nDied[^\n]*\.\nUNITCHECK failed[^\n]*\.\nunitcheck\ncheck\nend\z)/
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; UNITCHECK { die; }
# STATUS​: 65280
# Failed test 16 - CHECK{exit 1} should exit at ./test.pl line 1062
# got "begin\nunitcheck\ncheck"
# expected "begin\nunitcheck\ncheck\nend"
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; CHECK { exit 1; }
# STATUS​: 256
# Failed test 17 - CHECK{die} should exit at ./test.pl line 1074
# got 'begin\nunitcheck\nDied at - line 1.\nCHECK failed--call queue aborted.\ncheck'
# expected /(?^​:\Abegin\nunitcheck\nDied[^\n]*\.\nCHECK failed[^\n]*\.\ncheck\nend\z)/
# PROG​:
# BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; CHECK { die; }
# STATUS​: 65280
op/blocks.t ..
1..22
ok 1 - Order of execution of special blocks
ok 2 - blocks interact with packages/scopes
ok 3 - can name blocks as sub FOO
ok 4 - eval-UNITCHECK-eval (bug 70614)
ok 5 - constant named after a special block
ok 6 - context
ok 7 - null PL_curcop in newGP
ok 8 - blocks execute in right order
ok 9 - BEGIN{exit 0} doesn't exit yet
not ok 10 - BEGIN{exit 1} should exit
not ok 11 - BEGIN{die} should exit
ok 12 - UNITCHECK{exit 0} doesn't exit yet
not ok 13 - UNITCHECK{exit 1} should exit
not ok 14 - UNITCHECK{die} should exit
ok 15 - CHECK{exit 0} doesn't exit yet
not ok 16 - CHECK{exit 1} should exit
not ok 17 - CHECK{die} should exit
ok 18 - INIT{exit 0} should exit
ok 19 - INIT{exit 1} should exit
ok 20 - INIT{die} should exit
not ok 21 - RT \#2917​: No constraint on how late INIT blocks can run # TODO RT #2917​: INIT{} in eval is wrongly considered too late
# Failed test 21 - RT \#2917​: No constraint on how late INIT blocks can run at ./test.pl line 1062
# got ""
# expected "in init"
# PROG​:
# eval "INIT { print qq(in init); };";
# STATUS​: 0
ok 22 - RT \#113934​: goto out of BEGIN causes assertion failure
Failed 6/22 subtests

Test Summary Report


op/blocks.t (Wstat​: 0 Tests​: 22 Failed​: 6)
  Failed tests​: 10-11, 13-14, 16-17
Files=1, Tests=22, 7 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result​: FAIL

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @iabyn

On Wed, May 09, 2018 at 01​:15​:47PM +0100, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd reverted
passes all tests except t/op/blocks.t tests 10, 11, 13, 14, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains
the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant
tests under win32 etc.

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

@p5pRT
Copy link
Author

p5pRT commented May 9, 2018

From @xsawyerx

On 05/09/2018 11​:48 AM, Dave Mitchell wrote​:

On Tue, May 08, 2018 at 05​:57​:10PM +0100, Steve Hay via perl5-porters wrote​:

Pre-8e920bd341 outputs this​:
...
END​: parent​: killing (-12584)

Blead and your branch both output this​:
[...] Ctrl+C to kill it

Thanks for that, I finally know what's going on.

TL;DR​: a can of worms has been opened, and for 5.28 we should close
the can by reverting for now the 8e920bd PERL_EXIT_DESTRUCT_END
change (plus the tests), then re-address this issue post 5.28.

Thank you, Dave.

We should apply the fix that does this.

[...]

@p5pRT
Copy link
Author

p5pRT commented May 10, 2018

From @steve-m-hay

On 9 May 2018 at 14​:20, Dave Mitchell <davem@​iabyn.com> wrote​:

On Wed, May 09, 2018 at 01​:15​:47PM +0100, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd reverted
passes all tests except t/op/blocks.t tests 10, 11, 13, 14, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains
the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant
tests under win32 etc.

This passes (or skips...) all tests for me.

@p5pRT
Copy link
Author

p5pRT commented May 11, 2018

From @iabyn

On Thu, May 10, 2018 at 06​:03​:40PM +0100, Steve Hay via perl5-porters wrote​:

On 9 May 2018 at 14​:20, Dave Mitchell <davem@​iabyn.com> wrote​:

On Wed, May 09, 2018 at 01​:15​:47PM +0100, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd reverted
passes all tests except t/op/blocks.t tests 10, 11, 13, 14, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains
the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant
tests under win32 etc.

This passes (or skips...) all tests for me.

Now merged as v5.27.11-38-g64a9c78095.
I'll remove this ticket from the blockers list.

--
Red sky at night - gerroff my land!
Red sky at morning - gerroff my land!
  -- old farmers' sayings #14

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2019

From @jkeenan

On Fri, 11 May 2018 07​:46​:14 GMT, davem wrote​:

On Thu, May 10, 2018 at 06​:03​:40PM +0100, Steve Hay via perl5-porters
wrote​:

On 9 May 2018 at 14​:20, Dave Mitchell <davem@​iabyn.com> wrote​:

On Wed, May 09, 2018 at 01​:15​:47PM +0100, Steve Hay via perl5-
porters wrote​:

Current blead with 8e920bd
reverted
passes all tests except t/op/blocks.t tests 10, 11, 13, 14, 16 and
17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which
contains
the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the
relevant
tests under win32 etc.

This passes (or skips...) all tests for me.

Now merged as v5.27.11-38-g64a9c78095.
I'll remove this ticket from the blockers list.

OP reports that problem was fixed in perl-5.28. See​:
libwww-perl/LWP-Protocol-https#52 (comment)

Resolving ticket.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2019

@jkeenan - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Aug 11, 2019
@p5pRT p5pRT added BBC Blead Breaks CPAN - changes in blead broke a cpan module(s) Severity High labels Oct 19, 2019
@toddr toddr added this to the 5.32.0 milestone Oct 25, 2019
demerphq added a commit that referenced this issue Aug 29, 2022
…struct

See #16418

This should fix the issue with Win32.
demerphq added a commit that referenced this issue Aug 29, 2022
…struct

See #16418

This should fix the issue with Win32.
demerphq added a commit that referenced this issue Sep 3, 2022
…struct

See #16418

This should fix the issue with Win32.
scottchiefbaker pushed a commit to scottchiefbaker/perl5 that referenced this issue Nov 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s) distro-mswin32 type-core
Projects
None yet
Development

No branches or pull requests

2 participants