Perl CPAN Support, Frozen cpanm Command, Interactive Makefile.PL #1519

wbraswell · 2018-07-04T09:03:43Z

When I try to run fpm for the Inline::CPP software on CPAN, it gets partway through the process and then hangs. If you just look at the output, you will think it is waiting for info about Parse::RecDescent, but I put a bunch of checkpoints in the code and it is actually getting well past that.

$ fpm --no-cpan-test --cpan-verbose --verbose --debug-workspace --workdir /home/wbraswell/rperl_packager_tmp/ -t rpm -s cpan Inline::CPP
...
LOTS OF OUTPUT HERE
...
Installing /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan/lib/perl5/Test2/Formatter/TAP.pm {:level=>:info}
Appending installation info to /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan/lib/perl5/x86_64-linux-thread-multi/perllocal.pod {:level=>:info}
OK {:level=>:info}
Successfully installed Test-Simple-1.302136 (upgraded from 0.98) {:level=>:info}
Installing /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan/lib/perl5/x86_64-linux-thread-multi/.meta/Test-Simple-1.302136/install.json {:level=>:info}
Installing /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan/lib/perl5/x86_64-linux-thread-multi/.meta/Test-Simple-1.302136/MYMETA.json {:level=>:info}
<== Installed dependencies for /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/module. Finishing. {:level=>:info}
9 distributions installed {:level=>:info}
Asking metacpan about a module {:module=>"Carp", :level=>:info}
Asking metacpan about a module {:module=>"Inline", :level=>:info}
Asking metacpan about a module {:module=>"Inline::C", :level=>:info}
Asking metacpan about a module {:module=>"Parse::RecDescent", :level=>:info}

This is the line where it actually freezes:
https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L242-L246

This is the command which is actually frozen:

$ perl -Mlocal::lib=/home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan Makefile.PL PREFIX=/usr/local INSTALL_BASE=

If we run this command directly, we see it is interactive and is waiting on user input!

$ cd /home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/module
$ perl -Mlocal::lib=/home/wbraswell/rperl_packager_tmp/package-cpan-build-2cb511583273a27fa85f246094e26472cffc43fbd51a5581c969ecf4aa93/cpan Makefile.PL PREFIX=/usr/local INSTALL_BASE=
What C++ compiler would you like to use? [g++ -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -xc++] 
What default libraries would you like to include? [-lstdc++] 
Detected <iostream> style headers. ('.h' not needed.)
Generating a Unix-style Makefile
Writing Makefile for Inline::CPP
Writing MYMETA.yml and MYMETA.json

I'm not sure how to handle this just yet, let me know if you have any solutions, thanks! :-)

The text was updated successfully, but these errors were encountered:

liger1978 · 2018-07-04T09:24:42Z

I've worked this out I think.

See here and here.

It looks as if we need to set the PERL_MM_USE_DEFAULT environment variable to true.

liger1978 · 2018-07-04T14:45:52Z

I've put in a merge request that fixes this. @wbraswell You can test using my fork and branch if you like.

liger1978 · 2018-07-04T14:47:08Z

BTW I tried the environment variable approach but the variable was ignored. Forcing stdin to be enabled seemed to be the only effective way to prevent prompting.

wbraswell · 2018-07-04T17:02:05Z

@liger1978
Okay great, I will test it and let you know, thanks! :-)

wbraswell · 2018-07-04T17:43:08Z

@liger1978
Unfortunately it is still hanging for me at the exact same location using your new safesystemin() code...
This is the same command I am still using, can you please tell me if it works for you?

$ fpm --no-cpan-test --cpan-verbose --verbose --debug-workspace --workdir /home/wbraswell/rperl_packager_tmp/ -t rpm -s cpan Inline::CPP
Setting workdir {:workdir=>"/home/wbraswell/rperl_packager_tmp/", :level=>:info}
Asking metacpan about a module {:module=>"Inline::CPP", :level=>:info}
Downloading perl module {:distribution=>"Inline-CPP", :version=>nil, :level=>:info}
Setting package name from 'distribution' {:distribution=>"Inline-CPP", :level=>:info}
Installing any build or configure dependencies {:level=>:info}
Asking metacpan about a module {:module=>"Carp", :level=>:info}
Asking metacpan about a module {:module=>"Inline", :level=>:info}
Asking metacpan about a module {:module=>"Inline::C", :level=>:info}
Asking metacpan about a module {:module=>"Parse::RecDescent", :level=>:info}
***FROZEN HERE***

wbraswell · 2018-07-04T18:20:51Z

@liger1978
Okay I think I figured it out, in addition to the 1 location in cpan.rb where you have already replaced safesystem() with safesystemin(), we must also replace (at least) all the other places where safesystem() is called on cpanm, which are a total of 5 lines:

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L211

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L214

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L219

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L236

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L242

Considering you have already created your own fork and pull request, and my code is currently full of manual checkpoint debug statements, therefore I will wait for you to make these additional 5 replacements and update your pull request.

(If, for whatever reason, you want to cancel your own pull request, just let me know and I will clean up my code and make a pull request with all the changes to util.rb and cpan.rb needed to fix this issue.)

Thank you! :-)

liger1978 · 2018-07-04T18:30:38Z

I’ll have a look later today.

…

On 4 Jul 2018, at 19:21, William N. Braswell, Jr. ***@***.***> wrote: @liger1978 Okay I think I figured it out, in addition to the 1 location in cpan.rb where you have already replaced safesystem() with safesystemin(), we must also replace (at least) all the other places where safesystem() is called on cpanm, which are a total of 5 lines: https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L211 https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L214 https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L219 https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L236 https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/cpan.rb#L242 Considering you have already created your own fork and pull request, and my code is currently full of manual checkpoint debug statements, therefore I will wait for you to make these additional 5 replacements and update your pull request. (If, for whatever reason, you want to cancel your own pull request, just let me know and I will clean up my code and make a pull request with all the changes to util.rb and cpan.rb needed to fix this issue.) Thank you! :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

wbraswell · 2018-07-04T18:42:49Z

@liger1978
One other thing which you may be easy for you... For some modules, there is a long delay during configuration running safesystemin() on this line:
liger1978@97b4a65#diff-9c47944b0c3fe586b6ab1c7e3787d1b5R232

stdout_r_str = stdout.read

And after it is done executing this line successfully, we never see any of the actual STDOUT (or STDERR) which may have been produced, so in this example all of the output generated by the interactive configuration step of Inline::CPP is completely hidden from me, the developer. (This was part of the initial mystery of "why is this hanging?!?") I think this same hidden-output issue affects safesystem(), although it may be addressed in safesystemout()?

Can we somehow detect either fpm's own --verbose flag and/or the new --cpan-verbose flag I recently added, in order to unhide the output generated?

I am actually realizing now that this is very important, because during long complicated configure/build processes, I have no idea what is actually happening in the background w/out running ps aux and trying to just guess at what is going on...

wbraswell · 2018-07-04T18:57:40Z

Using my latest fork, which includes all new code from @liger1978 plus replacing the 5 cpanm calls with safesystemin() as described above, we are now hanging at a different place, on the same stdout_r_str = stdout.read line mentioned in the immediately-preceding comment.
https://github.com/wbraswell/fpm

$ fpm --no-cpan-test --cpan-verbose --verbose --debug-workspace --workdir /home/wbraswell/rperl_packager_tmp/ -t rpm -s cpan RPerl
Setting workdir {:workdir=>"/home/wbraswell/rperl_packager_tmp/", :level=>:info}
[[[ DEBUG CHECKPOINT 010 ]]] {:level=>:info}
Asking metacpan about a module {:module=>"RPerl", :level=>:info}
[[[ DEBUG CHECKPOINT 000 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 001 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 002 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 006 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 007 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 008 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 011 ]]] {:level=>:info}
Downloading perl module {:distribution=>"RPerl", :version=>nil, :level=>:info}
[[[ DEBUG CHECKPOINT 012 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 013 ]]] {:level=>:info}
Setting package name from 'distribution' {:distribution=>"RPerl", :level=>:info}
Installing any build or configure dependencies {:level=>:info}
[[[ DEBUG CHECKPOINT 060 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 100 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 101 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 120 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 121 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 122 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 123 ]]] {:level=>:info}
***FROZEN HERE***

After letting it run for a minute or two, it seems to be stopped on a call to as on XS.o:

$ ps ux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wbraswe+ 10171  0.0  0.0 128008  1824 ?        Ss   Apr15   0:11 SCREEN -R
wbraswe+ 10172  0.0  0.0 115392  2068 pts/1    Ss   Apr15   0:00 /bin/bash
wbraswe+ 14581  0.0  0.0 148684  2784 ?        S    10:28   0:00 sshd: wbraswell@pts/0
wbraswe+ 14582  0.0  0.0 115388  2108 pts/0    Ss   10:28   0:00 -bash
wbraswe+ 24308  0.4  0.4 234336 32172 pts/4    Sl+  11:53   0:00 ruby /home/wbraswell/repos_github/fpm-fork-latest/bin/fpm --no-cpan-test --cpan-verbose --verbose --debug-workspace --workdir /home/wbraswell/rperl_packager_tmp/ -t rpm -s cpan RPerl
wbraswe+ 24316  6.4  0.5 202804 41008 pts/4    S+   11:53   0:05 /usr/local/bin/cpanm
wbraswe+ 24634  0.0  0.0 128056  1812 ?        Ss   Jul01   0:02 SCREEN
wbraswe+ 24635  0.0  0.0 115536  2292 pts/2    Ss+  Jul01   0:00 /bin/bash
wbraswe+ 29227  0.0  0.0 128016  1820 ?        Ss   Jul03   0:07 SCREEN
wbraswe+ 29228  0.0  0.0 115624  2368 pts/4    Ss   Jul03   0:01 /bin/bash
wbraswe+ 29996  0.0  0.0 108320  1128 pts/4    S+   11:54   0:00 /usr/bin/make
wbraswe+ 30038  0.0  0.0 111792   944 pts/4    S+   11:54   0:00 gcc -c -I. -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -DVERSION="0.428" -DXS_VERSION="0.428" -fPIC -I/usr/lib64/perl5/CORE XS.c
wbraswe+ 30039  4.2  0.4 165068 32556 pts/4    S+   11:54   0:00 /usr/libexec/gcc/x86_64-redhat-linux/4.8.5/cc1 -quiet -I . -I /usr/local/include -I /usr/lib64/perl5/CORE -D _REENTRANT -D _GNU_SOURCE -D _LARGEFILE_SOURCE -D _FILE_OFFSET_BITS=64 -D VERSION="0.428" -D XS_VERSION="0.428" -D_FORTIFY_SOURCE=2 XS.c -quiet -dumpbase XS.c -m64 -mtune=generic -march=x86-64 -auxbase XS -g -grecord-gcc-switches -O2 -Wall -fno-strict-aliasing -fstack-protector -fexceptions -fstack-protector-strong -fPIC --param ssp-buffer-size=4 -o -
wbraswe+ 30040  0.0  0.0 125276  6344 pts/4    S+   11:54   0:00 as -I . -I /usr/local/include -I /usr/lib64/perl5/CORE --64 -o XS.o

I let it run for 12 minutes and it was still frozen on the call to as -I . -I /usr/local/include -I /usr/lib64/perl5/CORE --64 -o XS.o.

wbraswell · 2018-07-04T21:10:10Z

There are actually 2 different issues here, both related to cpanm freezing/hanging.
This issue 1519 is due to interactive build processes waiting for user input, solved by creating safesystemin() and handling STDIN properly.
The other issue 1522 is due to the STDOUT & STDERR being somehow handled improperly, which causes gcc and/or as to freeze in the background for unknown reason(s).
#1522

liger1978 · 2018-07-04T21:21:16Z

I've pushed again to https://github.com/liger1978/fpm/tree/1519 which adds the extra required safesystemin() calls in cpan.rb.

wbraswell · 2018-07-04T21:34:43Z

@jordansissel
In your latest push, you are still missing safesystemin() in at least 3 locations where an interactive configure process may occur:

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L212

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L215

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L220

It may also be smart to use safesystemin() during the build/make processes as well:

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L226

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L229

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L250

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L251

https://github.com/liger1978/fpm/blob/c3d7ae0d6d8f3a95d4189676c31ffc141f124cec/lib/fpm/package/cpan.rb#L252

liger1978 · 2018-07-04T21:54:38Z

I only included it in cases where MakeMaker would be used (which is what uses prompt).

wbraswell · 2018-07-04T21:58:18Z

Okay, is it possible that Build.PL will ever have an interactive configure or build process?

liger1978 · 2018-07-04T22:02:10Z

Don't know, but haven't seen it. MakeMaker (i.e. what processes Makefile.PL) is what has the prompt function.

wbraswell · 2018-07-04T22:06:13Z

I understand. My own fork has safesystemin() for both Build.PL & Makefile.PL, so that is what I am currently testing against.

liger1978 · 2018-07-04T22:06:44Z

BTW I've always had to use the --no-cpan-sandbox-non-core argument to have success with converting cpan modules. Don't know if you want to try this...

wbraswell · 2018-07-04T22:25:09Z

I have not needed to use the --no-cpan-sandbox-non-core option so far...

wbraswell · 2018-07-04T22:48:15Z

Regarding Build.PL and prompting...
https://metacpan.org/pod/Module::Build#installdeps

installdeps
[version 0.36]
This action will use the cpan_client parameter as a command to install missing prerequisites.
You will be prompted whether to install optional dependencies.

liger1978 · 2018-07-04T23:02:08Z

OK, but I think this is different. Build.PL only prompts if one supplies the cpan_client parameter (cpan.rb does not). Also, my safesystemin hack would not work anyway. It only works for Makefile.PL builds because ExtUtils::MakeMaker suppresses prompts in the presence of an empty stdin.

wbraswell · 2018-07-04T23:03:43Z

Okay I see.

This prevents subprocesses from mistakenly getting stuck trying to read from a stdin that won't say anything ;) Fixes #1519.

This prevents subprocesses from mistakenly getting stuck trying to read from a stdin that won't say anything ;) Fixes #1519 and #1522.

wbraswell · 2022-11-14T17:38:27Z

This issue is solved by PR #1955 by @NicholasBHubbard, great job everyone! :-)

jordansissel · 2022-11-15T16:31:57Z

fpm 1.15.0 is released and contains a fix for this issue :)

liger1978 added a commit to liger1978/fpm that referenced this issue Jul 4, 2018

Fixes jordansissel#1519 cpan: prevent MakeMaker user prompts

97b4a65

wbraswell mentioned this issue Jul 4, 2018

Fixes #1519 cpan: prevent MakeMaker user prompts #1520

Closed

wbraswell mentioned this issue Jul 4, 2018

Perl CPAN Support, Frozen cpanm Command, Broken I/O Handling In execmd() #1522

Closed

liger1978 added a commit to liger1978/fpm that referenced this issue Jul 4, 2018

Fixes jordansissel#1519 cpan: prevent MakeMaker user prompts

c3d7ae0

This was referenced Jul 10, 2018

Perl CPAN Support, Fix All 5 Outstanding Issues #1527

Closed

Perl CPAN Support, Fix All 5 Outstanding Issues, Second Try #1528

Closed

jordansissel added a commit that referenced this issue Feb 23, 2019

Always close stdin when running without a block.

f079a99

This prevents subprocesses from mistakenly getting stuck trying to read from a stdin that won't say anything ;) Fixes #1519.

jordansissel added a commit that referenced this issue Feb 23, 2019

Always close stdin when running without a block.

ae9594e

This prevents subprocesses from mistakenly getting stuck trying to read from a stdin that won't say anything ;) Fixes #1519 and #1522.

jordansissel mentioned this issue Feb 23, 2019

Fix a few CPAN issues #1610

Closed

NicholasBHubbard mentioned this issue Oct 25, 2022

Fix frozen cpanm caused by interactive Makefile.PL #1944

Closed

NicholasBHubbard mentioned this issue Nov 6, 2022

Execmd: prevent prompting for input #1955

Merged

wbraswell closed this as completed Nov 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perl CPAN Support, Frozen cpanm Command, Interactive Makefile.PL #1519

Perl CPAN Support, Frozen cpanm Command, Interactive Makefile.PL #1519

wbraswell commented Jul 4, 2018 •

edited

Loading

liger1978 commented Jul 4, 2018

liger1978 commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018 via email

wbraswell commented Jul 4, 2018 •

edited

Loading

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018 •

edited

Loading

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018 •

edited

Loading

wbraswell commented Jul 4, 2018

wbraswell commented Nov 14, 2022

jordansissel commented Nov 15, 2022

Perl CPAN Support, Frozen cpanm Command, Interactive Makefile.PL #1519

Perl CPAN Support, Frozen cpanm Command, Interactive Makefile.PL #1519

Comments

wbraswell commented Jul 4, 2018 • edited Loading

liger1978 commented Jul 4, 2018

liger1978 commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018 via email

wbraswell commented Jul 4, 2018 • edited Loading

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018 • edited Loading

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018

wbraswell commented Jul 4, 2018

wbraswell commented Jul 4, 2018

liger1978 commented Jul 4, 2018 • edited Loading

wbraswell commented Jul 4, 2018

wbraswell commented Nov 14, 2022

jordansissel commented Nov 15, 2022

wbraswell commented Jul 4, 2018 •

edited

Loading

wbraswell commented Jul 4, 2018 •

edited

Loading

wbraswell commented Jul 4, 2018 •

edited

Loading

liger1978 commented Jul 4, 2018 •

edited

Loading