Bleadperl breaks OALDERS/LWP-Protocol-https-6.07.tar.gz on Windows

[p5pRT]

Migrated from rt.perl.org#132863 (status was 'resolved')

Searchable as RT132863$

[p5pRT]

From @chorny

Created by @chorny

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: libwww-perl/LWP-Protocol-https#52

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.27.8:

Configured by strawberry-perl at Tue Feb 13 01:56:34 2018.

Summary of my perl5 (revision 5 version 27 subversion 8) configuration:

  Platform:
    osname=MSWin32
    osvers=5.1.2600
    archname=MSWin32-x86-multi-thread-64int
    uname='Win32 strawberry-perl 5.27.8.1-beta1 #1 Tue Feb 13 01:53:53
2018 i386'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -D__USE_MINGW_ANSI_STDIO
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=4
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=12
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++.exe'
    ldflags ='-s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"'
    libpth=H:\perl\278\c\lib H:\perl\278\c\i686-w64-mingw32\lib
H:\perl\278\c\lib\gcc\i686-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl527.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"'



@INC for perl 5.27.8:
    H:/perl/278/perl/site/lib
    H:/perl/278/perl/vendor/lib
    H:/perl/278/perl/lib


Environment for perl 5.27.8:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=H:\perl\278\perl\site\bin;H:\perl\278\perl\bin;H:\perl\278\c\bin;C:\Program
Files\Far\;C:\Prg\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
Files\ATI Technologies\ATI.ACE\Core-Static;C:\strawberry514\c\bin;C:\strawberry514\perl\site\bin;C:\strawberry514\perl\bin;C:\Program
Files\TortoiseHg\;C:\Program Files\MySQL\MySQL Server
5.5\bin;c:\util\;C:\Prg\Subversion\bin;C:\Program
Files\TortoiseSVN\bin;D:\vagrant\vagrant\bin;C:\Prg\TortoiseGit\bin;C:\Prg\Git\cmd;C:\Program
Files\Skype\Phone\;
    PERL_BADLANG (unset)
    SHELL (unset)


-- 
Alexandr Ciornii, http://chorny.net

[p5pRT]

From @jkeenan

On Tue, 13 Feb 2018 11​:43​:20 GMT, chorny wrote​:

This is a bug report for perl from alexchorny@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.27.8.

-----------------------------------------------------------------
[Please describe your issue here]

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: https://github.com/libwww-perl/LWP-Protocol-
https/issues/52

I see that the test file which is hanging has code in it like this​:

#####
defined( my $pid = fork()) or die "fork failed​: $!";

# child process runs _server and exits
if ( ! $pid ) {
  @​childs = ();
  exit( _server());
}
#####

... and as well​:

#####
$ grep -n goto t/https_proxy.t
205​: my $cl = $server->accept or goto ACCEPT;
215​: goto ACCEPT;
230​: goto ACCEPT; # wait for next connection
285​: goto SSL_UPGRADE;
306​: goto REQUEST if $keep_alive;
#####

'goto' is cited in commit messages for the following commits in the last 3 months​:

#####
436908e
b537774
e7afb05
a01f464
e57923a
da4e040
e57923a
6d90e98
4d7e83b
4bfb553
#####

Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

List​: Do people on other versions of Windows also get this hang?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @chorny

2018-02-14 16​:56 GMT+02​:00 James E Keenan via RT <perlbug-followup@​perl.org>​:

On Tue, 13 Feb 2018 11​:43​:20 GMT, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.
Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings". Commit 503bc07
does not have such problems.
Other affected distributions are Test-WWW-Mechanize-Catalyst and
Test-HTTP-Server-Simple and possibly others.

When installing IO​::Socket​::SSL, you should install
MIKEM/Net-SSLeay-1.82.tar.gz first and skip it's test
t/local/33_x509_create_cert.t.

--
Alexandr Ciornii, http​://chorny.net

[p5pRT]

From @chorny

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

--
Alexandr Ciornii, http​://chorny.net

[p5pRT]

From [Unknown Contact. See original ticket]

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1, 5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

--
Alexandr Ciornii, http​://chorny.net

[p5pRT]

From @jkeenan

On Sat, 21 Apr 2018 22​:51​:22 GMT, chorny wrote​:

On Tue, 13 Feb 2018 03​:43​:20 -0800, chorny wrote​:

https_proxy.t hangs on 5.27.8, did not hang on 5.27.6, 5.27.1,
5.26.0.
All tests of https_proxy.t pass, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz
(https://rt.cpan.org/Ticket/Display.html?id=125169)

It appears that we neglected to add this RT to the list of perl-5.28.0 blockers (RT 127689). I have done so now.

These failures appear to be Windows-specific. Just now I used cpanm to successfully install every module listed in this RT on both Linux and FreeBSD-11.1.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @iabyn

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

  END { kill 9,@​childs if @​childs };

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

Do you have any insight?

--
Dave's first rule of Opera​:
If something needs saying, say it​: don't warble it.

[p5pRT]

From @iabyn

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@​childs if @​childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

  LWP-Protocol-https-6.0
  KAORU/HTTP-Async-0.33
  IO​::Socket​::SSL

And ideally confirm my diagnosis that on fork() on win32, END()
blocks were being called twice in the child (and presumably once in the
parent), but only once after my patch. If someone could write a test for
this, that would be even better,

Here's the commit message​:

  win323 fork()​: honour PERL_EXIT_DESTRUCT_END
 
  RT #132863
 
  The PERL_EXIT_DESTRUCT_END flag in PL_exit_flags is designed to defer
  the calling of END blocks in perl_run() to being called from
  perl_destruct() instead. On UNIX-like builds, perlmain.c sets this flag.
  So main() looks like, in outline​:
 
  PL_exit_flags |= PERL_EXIT_DESTRUCT_END;
  if (!perl_parse(my_perl, xs_init, argc, argv, (char **)NULL))
  perl_run(my_perl);
  exitstatus = perl_destruct(my_perl);
 
  which means that it doesn't matter whether perl_parse() finishes
  normally or prematurely (e.g. via BEGIN { exit(1) } or BEGIN { die });
  in all cases, due to PERL_EXIT_DESTRUCT_END being set, the END blocks
  will always be called (from perl_destruct()).
 
  Commit v5.27.7-9-g8e920bd341 added PERL_EXIT_DESTRUCT_END to the
  equivalent of main() on other platforms such as win32; this
  means that the new tests in t/op/blocks.t pass on win32 too, and all
  platforms have the same behaviour for e.g.
 
  END { print "end\n"; } BEGIN { exit 1; } # prints "end"
 
  However, that commit was causing some tests in some CPAN distributions to
  hang. These were using the win32 fork() emulation.
 
  PerlProcFork() on win32 clones an interpreter, starts a new thread,
  and makes the new thread call win32_start_child(), which
  does the rough equivalent of C<perl_run(); perl_destruct();>,
  except that it rolls it's own perl_run() equivalent which
  *doesn't* honour the PERL_EXIT_DESTRUCT_END flag. So by setting
  PERL_EXIT_DESTRUCT_END, END blocks in fork()ed processes were getting
  executed twice​: once by win32_start_child() emulating perl_run(),
  and once by perl_destruct() called from win32_start_child().
 
  This commit makes win32_start_child() honour PERL_EXIT_DESTRUCT_END.

--
It's not that I'm afraid to die, I just don't want to be there when it
happens.
  -- Woody Allen

[p5pRT]

From @steve-m-hay

On 30 April 2018 at 22​:15, Dave Mitchell <davem@​iabyn.com> wrote​:

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@​childs if @​childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO​::Socket​::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs
before doing anything. With your branch it now passes all 8 tests but
then hangs instead of exiting. So it's an improvement of sorts, but
something is still amiss somewhere. However, there's no improvement
with t/session_ticket.t. With either build I get the following
failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t
1..6
# listen at 127.0.0.1​:51973
# listen at 127.0.0.1​:51974
access to server[0]
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 1 - no initial session -> no reuse
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 2 - reuse with the next session and secret[0]
access to server[1]
using current ticket secret
server[1] reused=1
rotate secrets
# connect to 1​: success reuse=1
ok 3 - reuse even though server changed, since they share ticket secret
access to server[1]
using non-current ticket secret
creating new ticket key2
server[1] reused=0
rotate secrets
# connect to 1​: success reuse=0
ok 4 - reports non-reuse since server1 changed secret to secret[1]
access to server[0]
using non-current ticket secret
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 6 - reuse again since got ticket with secret[0] in last step
not ok 1 - select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
#
# Failed test 'select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
# '
# at ./t/testlib.pl line 39.

I haven't tried an earlier
(pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that
this used to work. I will try to do that in the next day or two.

[p5pRT]

From @steve-m-hay

On 3 May 2018 at 18​:20, Steve Hay <steve.m.hay@​googlemail.com> wrote​:

On 30 April 2018 at 22​:15, Dave Mitchell <davem@​iabyn.com> wrote​:

On Mon, Apr 23, 2018 at 01​:39​:27PM +0100, Dave Mitchell wrote​:

On Thu, Mar 08, 2018 at 01​:06​:24AM +0200, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs -
IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails.
Problem appear starting with commit
8e920bd - "set PERL_EXIT_DESTRUCT_END
in all embeddings".

Zefram, that commit was one of yours. I don't quite understand the
implications of it, but note that the test file which newly hangs on
Windows contains both a fork() and

END \{ kill 9\,@​childs if @​childs \};

so perhaps an END block wasn't being called before, or used to be called
only once, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit, which may fix this.
Note that I've made the change in win32/perlhost.h blind, as I haven't
tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32;
(b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO​::Socket​::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs
before doing anything. With your branch it now passes all 8 tests but
then hangs instead of exiting. So it's an improvement of sorts, but
something is still amiss somewhere. However, there's no improvement
with t/session_ticket.t. With either build I get the following
failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t
1..6
# listen at 127.0.0.1​:51973
# listen at 127.0.0.1​:51974
access to server[0]
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 1 - no initial session -> no reuse
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 2 - reuse with the next session and secret[0]
access to server[1]
using current ticket secret
server[1] reused=1
rotate secrets
# connect to 1​: success reuse=1
ok 3 - reuse even though server changed, since they share ticket secret
access to server[1]
using non-current ticket secret
creating new ticket key2
server[1] reused=0
rotate secrets
# connect to 1​: success reuse=0
ok 4 - reports non-reuse since server1 changed secret to secret[1]
access to server[0]
using non-current ticket secret
creating new ticket key1
server[0] reused=0
# connect to 0​: success reuse=0
ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step
access to server[0]
using current ticket secret
server[0] reused=1
# connect to 0​: success reuse=1
ok 6 - reuse again since got ticket with secret[0] in last step
not ok 1 - select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
#
# Failed test 'select failed or timed out​: Bad file descriptor at
t\session_ticket.t line 113.
# '
# at ./t/testlib.pl line 39.

I haven't tried an earlier
(pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that
this used to work. I will try to do that in the next day or two.

I've now tried a build of the change prior to
8e920bd (namely,
503bc07) and unfortunately both
t/mitm.t and t/session_ticket.t pass all tests (and don't hang).
That's on the same machine with the same compiler and build
configuration.

So the problem here isn't yet fixed.

[p5pRT]

From @iabyn

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

  $| = 1;
  my $pid = fork();
  die "fork​: $!\n" unless defined $pid;
  my $who = $pid ? "parent" : "child";
  END { print "END​: $who\n" }

--
Overhead, without any fuss, the stars were going out.
  -- Arthur C Clarke

[p5pRT]

From @steve-m-hay

On 4 May 2018 at 20​:39, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork​: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END​: $who\\n" \}

They all output​:

END​: parent

[p5pRT]

From @iabyn

On Fri, May 04, 2018 at 09​:36​:17PM +0100, Steve Hay via perl5-porters wrote​:

On 4 May 2018 at 20​:39, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, May 04, 2018 at 05​:58​:50PM +0100, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on
1) something pre v5.27.7-9-g8e920bd341
2) blead
3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork​: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END​: $who\\n" \}

They all output​:

END​: parent

Hmmm.

Could you show the output from the attached script for each of the perls
described above, thanks?

--
I before E. Except when it isn't.

[p5pRT]

From @iabyn

#!/usr/bin/perl

use warnings;
use strict;

my ($pid, @​pid);
my $type = 'parent';

END {
  print STDERR "END​: $type​: killing (@​pid)\n";
  kill 9,@​pid;
  print STDERR "END​: $type​: done kill (@​pid)\n";
}

$pid = fork;
die $! unless defined $pid;
if (!$pid) {
  $type = 'child';
  while (1) {
  print STDERR "child​: sleep(1)\n";
  sleep(1);
  }
  print STDERR "child​: exit(1)\n";
  exit(1);
}

push @​pid, $pid;
print STDERR "parent​: sleep(4)\n";
sleep(4);
print STDERR "parent​: finishing\n";