Perl Modules #186

Closed
genio opened this Issue Apr 13, 2016 · 40 comments

Comments

Projects
None yet
@genio

genio commented Apr 13, 2016

Installing Perl modules is an essential part of working with Perl. Rather than get into an in-depth explanation of using cpan or cpanm along with local::lib, I'll just break it down into the simplest way possible, manual installation.

root@localhost:~# curl -LO https://cpan.metacpan.org/authors/id/E/ET/ETHER/Try-Tiny-0.24.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30666  100 30666    0     0   100k      0 --:--:-- --:--:-- --:--:--  100k
root@localhost:~# tar zxvf Try-Tiny-0.24.tar.gz
Try-Tiny-0.24/
Try-Tiny-0.24/Changes
Try-Tiny-0.24/CONTRIBUTING
Try-Tiny-0.24/dist.ini
Try-Tiny-0.24/INSTALL
Try-Tiny-0.24/lib/
Try-Tiny-0.24/LICENCE
Try-Tiny-0.24/maint/
Try-Tiny-0.24/Makefile.PL
Try-Tiny-0.24/MANIFEST
Try-Tiny-0.24/META.json
Try-Tiny-0.24/META.yml
Try-Tiny-0.24/README
Try-Tiny-0.24/t/
Try-Tiny-0.24/xt/
Try-Tiny-0.24/xt/author/
Try-Tiny-0.24/xt/release/
Try-Tiny-0.24/xt/release/changes_has_content.t
Try-Tiny-0.24/xt/release/cpan-changes.t
Try-Tiny-0.24/xt/release/distmeta.t
Try-Tiny-0.24/xt/release/pod-no404s.t
Try-Tiny-0.24/xt/release/portability.t
Try-Tiny-0.24/xt/author/00-compile.t
Try-Tiny-0.24/xt/author/eol.t
Try-Tiny-0.24/xt/author/kwalitee.t
Try-Tiny-0.24/xt/author/mojibake.t
Try-Tiny-0.24/xt/author/no-tabs.t
Try-Tiny-0.24/xt/author/pod-coverage.t
Try-Tiny-0.24/xt/author/pod-spell.t
Try-Tiny-0.24/xt/author/pod-syntax.t
Try-Tiny-0.24/t/00-report-prereqs.dd
Try-Tiny-0.24/t/00-report-prereqs.t
Try-Tiny-0.24/t/basic.t
Try-Tiny-0.24/t/context.t
Try-Tiny-0.24/t/erroneous_usage.t
Try-Tiny-0.24/t/finally.t
Try-Tiny-0.24/t/given_when.t
Try-Tiny-0.24/t/global_destruction_forked.t
Try-Tiny-0.24/t/global_destruction_load.t
Try-Tiny-0.24/t/lib/
Try-Tiny-0.24/t/named.t
Try-Tiny-0.24/t/when.t
Try-Tiny-0.24/t/lib/TryUser.pm
Try-Tiny-0.24/maint/bench.pl
Try-Tiny-0.24/lib/Try/
Try-Tiny-0.24/lib/Try/Tiny.pm
root@localhost:~# cd Try-Tiny-0.24
root@localhost:~/Try-Tiny-0.24# ll
total 136
drwx------ 2  502 staff     0 Dec 11 05:20 ./
drwx------ 2 root root      0 Apr 13 23:35 ../
-rw------- 1  502 staff  2885 Dec 11 05:20 Changes
-rw-r--r-- 1  502 staff  3607 Dec 11 05:20 CONTRIBUTING
-rw------- 1  502 staff   837 Dec 11 05:20 dist.ini
-rw-r--r-- 1  502 staff   925 Dec 11 05:20 INSTALL
drwx------ 2  502 staff     0 Apr 13 23:35 lib/
-rw-r--r-- 1  502 staff  1182 Dec 11 05:20 LICENCE
drwx------ 2  502 staff     0 Apr 13 23:35 maint/
-rw-r--r-- 1  502 staff  1435 Dec 11 05:20 Makefile.PL
-rw-r--r-- 1  502 staff   714 Dec 11 05:20 MANIFEST
-rw-r--r-- 1  502 staff 41924 Dec 11 05:20 META.json
-rw-r--r-- 1  502 staff 24356 Dec 11 05:20 META.yml
-rw-r--r-- 1  502 staff   332 Dec 11 05:20 README
drwx------ 2  502 staff     0 Apr 13 23:35 t/
drwx------ 2  502 staff     0 Dec 11 05:20 xt/
root@localhost:~/Try-Tiny-0.24# perl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
        lib/Try/Tiny.pm
        maint/bench.pl
        t/00-report-prereqs.dd
        t/00-report-prereqs.t
        t/basic.t
        t/context.t
        t/erroneous_usage.t
        t/finally.t
        t/given_when.t
        t/global_destruction_forked.t
        t/global_destruction_load.t
        t/lib/TryUser.pm
        t/named.t
        t/when.t
        xt/author/00-compile.t
        xt/author/eol.t
        xt/author/kwalitee.t
        xt/author/mojibake.t
        xt/author/no-tabs.t
        xt/author/pod-coverage.t
        xt/author/pod-spell.t
        xt/author/pod-syntax.t
        xt/release/changes_has_content.t
        xt/release/cpan-changes.t
        xt/release/distmeta.t
        xt/release/pod-no404s.t
        xt/release/portability.t
Please inform the author.
Writing Makefile for Try::Tiny
Wide character in print at /usr/share/perl/5.18/ExtUtils/MakeMaker.pm line 1034.
Wide character in print at /usr/share/perl/5.18/ExtUtils/MakeMaker.pm line 1034.
Wide character in print at /usr/share/perl/5.18/ExtUtils/MakeMaker.pm line 1034.
Writing MYMETA.yml and MYMETA.json
root@localhost:~/Try-Tiny-0.24#

In short, Perl modules cannot be installed because, for some reason, the files aren't really there even though I can browse, view, edit, delete, etc. those files.

@sunilmut

This comment has been minimized.

Show comment
Hide comment
@sunilmut

sunilmut Apr 14, 2016

Member

Thanks for trying this out. Could you collect a strace for the 'perl Makefile.PL' command and see why it is complaining?

Member

sunilmut commented Apr 14, 2016

Thanks for trying this out. Could you collect a strace for the 'perl Makefile.PL' command and see why it is complaining?

@genio

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge Apr 16, 2016

What version of ExtUtils::MakeMaker is installed?

What version of ExtUtils::MakeMaker is installed?

@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge Apr 16, 2016

...and what version of ExtUtils::Manifest?

and then, can you try again with updated EUM+EUMM and see if the outcome is different/better?

...and what version of ExtUtils::Manifest?

and then, can you try again with updated EUM+EUMM and see if the outcome is different/better?

@kentfredric

This comment has been minimized.

Show comment
Hide comment
@kentfredric

kentfredric Apr 16, 2016

Just commenting to keep microsoft updated on the issue so it doesn't look like a tumbleweed.

Issue has been reproduced using Perl 5.18.2's File::Find ( which there is no upgrade path for without an upgrade of Perl itself ).

root@localhost:~/Try-Tiny-0.24# perl -MFile::Find -E 'say find({wanted=>sub{warn $File::Find::name}}, q(.))'

^ Shows only top level entries, fails to recurse.

root@localhost:~/Try-Tiny-0.24# find

^ However shows entries recursively.

kentfredric commented Apr 16, 2016

Just commenting to keep microsoft updated on the issue so it doesn't look like a tumbleweed.

Issue has been reproduced using Perl 5.18.2's File::Find ( which there is no upgrade path for without an upgrade of Perl itself ).

root@localhost:~/Try-Tiny-0.24# perl -MFile::Find -E 'say find({wanted=>sub{warn $File::Find::name}}, q(.))'

^ Shows only top level entries, fails to recurse.

root@localhost:~/Try-Tiny-0.24# find

^ However shows entries recursively.

@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Apr 16, 2016

Perl 5.18.2
ExtUtils::MakeMaker 6.66

I downloaded 7.1 and ran the following with the same result:

perl -I/root/ExtUtils-MakeMaker-7.10/lib Makefile.PL

This, however, shows all of the files:

perl -MFile::Find -E '$File::Find::dont_use_nlink=1; say find({wanted=>sub{warn $File::Find::name},no_chdir=>1}, q(.))

genio commented Apr 16, 2016

Perl 5.18.2
ExtUtils::MakeMaker 6.66

I downloaded 7.1 and ran the following with the same result:

perl -I/root/ExtUtils-MakeMaker-7.10/lib Makefile.PL

This, however, shows all of the files:

perl -MFile::Find -E '$File::Find::dont_use_nlink=1; say find({wanted=>sub{warn $File::Find::name},no_chdir=>1}, q(.))
@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Apr 16, 2016

I forgot to mention that the version of ExtUtils::Manifest is 1.63

genio commented Apr 16, 2016

I forgot to mention that the version of ExtUtils::Manifest is 1.63

@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Apr 16, 2016

Well, doing the following will allow users to install modules:

  1. apt-get install liblocal-lib-perl cpanminus build-essential sed
  2. Turn off use of nlink:
    • sed -ri '1113i $File::Find::dont_use_nlink=1;' /usr/share/perl/5.18/File/Find.pm
    • OR
    • sed -ri 's/dont_use_nlink => .*?,$/dont_use_nlink => 1,/' /usr/lib/perl/5.18.2/Config.pm
  3. eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
  4. cpanm Try::Tiny

This is just a work-around.

genio commented Apr 16, 2016

Well, doing the following will allow users to install modules:

  1. apt-get install liblocal-lib-perl cpanminus build-essential sed
  2. Turn off use of nlink:
    • sed -ri '1113i $File::Find::dont_use_nlink=1;' /usr/share/perl/5.18/File/Find.pm
    • OR
    • sed -ri 's/dont_use_nlink => .*?,$/dont_use_nlink => 1,/' /usr/lib/perl/5.18.2/Config.pm
  3. eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
  4. cpanm Try::Tiny

This is just a work-around.

@yuki-kimoto

This comment has been minimized.

Show comment
Hide comment
@yuki-kimoto

yuki-kimoto Aug 17, 2016

Latest Perl resolve File::Find problem?

Or problem is the implementation of hard link in "bash on windows" ?

Latest Perl resolve File::Find problem?

Or problem is the implementation of hard link in "bash on windows" ?

@cthulhuology

This comment has been minimized.

Show comment
Hide comment
@cthulhuology

cthulhuology Aug 17, 2016

A better patch than editing /user/share/perl/5.18/File/Find.pm is to change line 94 in /usr/lib/perl/5.18.2/Config.pm to:
dont_use_nlink => 1,

cthulhuology commented Aug 17, 2016

A better patch than editing /user/share/perl/5.18/File/Find.pm is to change line 94 in /usr/lib/perl/5.18.2/Config.pm to:
dont_use_nlink => 1,

@yuki-kimoto

This comment has been minimized.

Show comment
Hide comment
@yuki-kimoto

yuki-kimoto Aug 18, 2016

This is great way! This is easy and better!

yuki-kimoto commented Aug 18, 2016

This is great way! This is easy and better!

@yuki-kimoto

This comment has been minimized.

Show comment
Hide comment
@yuki-kimoto

yuki-kimoto Aug 18, 2016

Improve the way to install cpan module.

  1. apt-get install liblocal-lib-perl cpanminus build-essentia
  2. vim /usr/lib/perl/5.18.2/Config.pm set dont_use_nlink => 1 line 94
  3. eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
  4. cpanm Try::Tiny

Improve the way to install cpan module.

  1. apt-get install liblocal-lib-perl cpanminus build-essentia
  2. vim /usr/lib/perl/5.18.2/Config.pm set dont_use_nlink => 1 line 94
  3. eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
  4. cpanm Try::Tiny
@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Aug 18, 2016

I'm not quite sure which is better. The first way doesn't result in requiring Config.pm at all. I'm not aware of anywhere else that uses the dont_use_nlink value within Config.pm other than File::Find. However, if other places are relying on that value and get their information from Config.pm, then maybe the second one is better.

That being said, I updated the work-around to show both methods. They're both a really easy one-line sed command.

genio commented Aug 18, 2016

I'm not quite sure which is better. The first way doesn't result in requiring Config.pm at all. I'm not aware of anywhere else that uses the dont_use_nlink value within Config.pm other than File::Find. However, if other places are relying on that value and get their information from Config.pm, then maybe the second one is better.

That being said, I updated the work-around to show both methods. They're both a really easy one-line sed command.

@yuki-kimoto

This comment has been minimized.

Show comment
Hide comment
@yuki-kimoto

yuki-kimoto Aug 18, 2016

I apply the above patch, and install some modules.

Installation of IO::Socket::IP 0.38 fail.

Checking dependencies from MYMETA.json ...
Checking if you have IO::Socket 0 ... Yes (1.36)
Checking if you have Socket 1.97 ... Yes (2.009)
Checking if you have Test::More 0.88 ... Yes (0.98)
Building and testing IO-Socket-IP-0.38
Building IO-Socket-IP
t/00use.t ........................... ok
t/01local-client-v4.t ............... ok
t/02local-server-v4.t ............... ok
t/03local-cross-v4.t ................ ok
t/04local-client-v6.t ............... ok
t/05local-server-v6.t ............... ok
t/06local-cross-v6.t ................ ok
t/10args.t .......................... ok
t/11sockopts.t ...................... ok
t/12port-fallback.t ................. ok
t/13addrinfo.t ...................... ok
t/14fileno.t ........................ ok
t/15io-socket.t ..................... ok
t/16v6only.t ........................ ok
t/17gai-flags.t ..................... ok
t/18fdopen.t ........................ ok
t/19no-addrs.t ...................... ok
t/20subclass.t ...................... ok
t/21as-inet.t ....................... ok
t/22timeout.t ....................... ok
t/30nonblocking-connect.t ........... ok

#   Failed test '$socket not yet connected'
#   at t/31nonblocking-connect-internet.t line 38.

#   Failed test '->connect eventually fails with ECONNREFUSED'
#   at t/31nonblocking-connect-internet.t line 92.
#   dollarbang = Transport endpoint is not connected
# Looks like you failed 2 tests of 9.
t/31nonblocking-connect-internet.t ..
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/9 subtests
t/99pod.t ........................... skipped: Test::Pod 1.00 required for testing POD

yuki-kimoto commented Aug 18, 2016

I apply the above patch, and install some modules.

Installation of IO::Socket::IP 0.38 fail.

Checking dependencies from MYMETA.json ...
Checking if you have IO::Socket 0 ... Yes (1.36)
Checking if you have Socket 1.97 ... Yes (2.009)
Checking if you have Test::More 0.88 ... Yes (0.98)
Building and testing IO-Socket-IP-0.38
Building IO-Socket-IP
t/00use.t ........................... ok
t/01local-client-v4.t ............... ok
t/02local-server-v4.t ............... ok
t/03local-cross-v4.t ................ ok
t/04local-client-v6.t ............... ok
t/05local-server-v6.t ............... ok
t/06local-cross-v6.t ................ ok
t/10args.t .......................... ok
t/11sockopts.t ...................... ok
t/12port-fallback.t ................. ok
t/13addrinfo.t ...................... ok
t/14fileno.t ........................ ok
t/15io-socket.t ..................... ok
t/16v6only.t ........................ ok
t/17gai-flags.t ..................... ok
t/18fdopen.t ........................ ok
t/19no-addrs.t ...................... ok
t/20subclass.t ...................... ok
t/21as-inet.t ....................... ok
t/22timeout.t ....................... ok
t/30nonblocking-connect.t ........... ok

#   Failed test '$socket not yet connected'
#   at t/31nonblocking-connect-internet.t line 38.

#   Failed test '->connect eventually fails with ECONNREFUSED'
#   at t/31nonblocking-connect-internet.t line 92.
#   dollarbang = Transport endpoint is not connected
# Looks like you failed 2 tests of 9.
t/31nonblocking-connect-internet.t ..
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/9 subtests
t/99pod.t ........................... skipped: Test::Pod 1.00 required for testing POD
@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Aug 18, 2016

@yuki-kimoto That module not installing is a different thing entirely. For module support, you should really take this discussion to IRC:
#perl on irc.perl.org

genio commented Aug 18, 2016

@yuki-kimoto That module not installing is a different thing entirely. For module support, you should really take this discussion to IRC:
#perl on irc.perl.org

@ryanerwin

This comment has been minimized.

Show comment
Hide comment
@ryanerwin

ryanerwin Dec 3, 2016

What is the Microsoft / Unbntu plan for this? Seems like the ideal think would be to make nlink work on WSL. In either case, I hope when I'm reinstalling or upgrading WSL I don't run into tweaking my /usr/lib/perl/5.18.2/Config.pm again...

What is the Microsoft / Unbntu plan for this? Seems like the ideal think would be to make nlink work on WSL. In either case, I hope when I'm reinstalling or upgrading WSL I don't run into tweaking my /usr/lib/perl/5.18.2/Config.pm again...

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 3, 2016

Actually the ideal situation would be that File::Find detects that nlink can't be used, and disables it for us. nlink in File::Find breaks with many file systems, including ISO9660. Which is why File::Find tries (and fails on WSL) to figure out whether nlink can safely be used, and disables it if not.

In a discussion from 2000, Randal L. Schwartz said:

"If you find an architecture on which you need to set this manually, please report it to the developers. This is supposed to be set automatically on those system which need it. If it isn't, it's a bug."
http://www.perlmonks.org/?node_id=32791

Ideal situation? File::Find figures out to disable nlink when running on WSL. Whether that requires a change to File::Find or to WSL I wouldn't hazard a guess.

It could be that File::Find's nlink detection is thrown off because it doesn't expect to be in a "Ubuntu on Windows" environment, in which case File::Find would need to change.

It could be that File::Find's nlink detection doesn't work because WSL delivers the wrong answers to the tests File::Find makes to see whether nlink is supported, in which case WSL would need to change.

Actually the ideal situation would be that File::Find detects that nlink can't be used, and disables it for us. nlink in File::Find breaks with many file systems, including ISO9660. Which is why File::Find tries (and fails on WSL) to figure out whether nlink can safely be used, and disables it if not.

In a discussion from 2000, Randal L. Schwartz said:

"If you find an architecture on which you need to set this manually, please report it to the developers. This is supposed to be set automatically on those system which need it. If it isn't, it's a bug."
http://www.perlmonks.org/?node_id=32791

Ideal situation? File::Find figures out to disable nlink when running on WSL. Whether that requires a change to File::Find or to WSL I wouldn't hazard a guess.

It could be that File::Find's nlink detection is thrown off because it doesn't expect to be in a "Ubuntu on Windows" environment, in which case File::Find would need to change.

It could be that File::Find's nlink detection doesn't work because WSL delivers the wrong answers to the tests File::Find makes to see whether nlink is supported, in which case WSL would need to change.

@ryanerwin

This comment has been minimized.

Show comment
Hide comment
@ryanerwin

ryanerwin Dec 3, 2016

@yorickdowne

I see your point. Looking at the File::Find code I see:

$File::Find::dont_use_nlink = 1
    if $^O eq 'os2' || $^O eq 'dos' || $^O eq 'amigaos' || $Is_Win32 ||
       $^O eq 'interix' || $^O eq 'cygwin' || $^O eq 'epoc' || $^O eq 'qnx' ||
     $^O eq 'nto';

Of course when I run: perl -e 'print $^O' under WSL, I get linux.

I had a similar problem in my own ~/.zshrc. There were some commands that I only wanted to run if I was running under WSL. Currently my solution is:

if [ -f "/mnt/c/Windows/System32/bash.exe" ]; then
  # must be running under WSL...
fi

The problem with this code is that someone may change their Windows folder to another location, but WSL doesn't have access to the windows environment variables to check for %WINDIR%.

Apparently some newer development versions of WSL support calling cmd.exe from inside WSL, so that may provide a better solution.

@yorickdowne

I see your point. Looking at the File::Find code I see:

$File::Find::dont_use_nlink = 1
    if $^O eq 'os2' || $^O eq 'dos' || $^O eq 'amigaos' || $Is_Win32 ||
       $^O eq 'interix' || $^O eq 'cygwin' || $^O eq 'epoc' || $^O eq 'qnx' ||
     $^O eq 'nto';

Of course when I run: perl -e 'print $^O' under WSL, I get linux.

I had a similar problem in my own ~/.zshrc. There were some commands that I only wanted to run if I was running under WSL. Currently my solution is:

if [ -f "/mnt/c/Windows/System32/bash.exe" ]; then
  # must be running under WSL...
fi

The problem with this code is that someone may change their Windows folder to another location, but WSL doesn't have access to the windows environment variables to check for %WINDIR%.

Apparently some newer development versions of WSL support calling cmd.exe from inside WSL, so that may provide a better solution.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 3, 2016

@ryanerwin Thanks for digging and finding that!

We can positively detect WSL via /proc/version and MS states they can't see a situation where they'd change that. See #423

Do you think checking for the presence of "Microsoft" or "WSL" in /proc/version just for nlink is the right way to go?
I wouldn't want to set Is_Win32 because it's not, by and large apps should think they are on Linux. Ubuntu on WSL is close enough to Linux for that to be the right way to go, though there's no actual Linux anywhere to be seen, just a bunch of GNU and other apps running on WSL :).

If /proc/version is the right way to go here, I can create a pull request with it for File::Find (assuming that lives on GitHub).

yorickdowne commented Dec 3, 2016

@ryanerwin Thanks for digging and finding that!

We can positively detect WSL via /proc/version and MS states they can't see a situation where they'd change that. See #423

Do you think checking for the presence of "Microsoft" or "WSL" in /proc/version just for nlink is the right way to go?
I wouldn't want to set Is_Win32 because it's not, by and large apps should think they are on Linux. Ubuntu on WSL is close enough to Linux for that to be the right way to go, though there's no actual Linux anywhere to be seen, just a bunch of GNU and other apps running on WSL :).

If /proc/version is the right way to go here, I can create a pull request with it for File::Find (assuming that lives on GitHub).

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 3, 2016

Okay I have something that works.

Edit: That code was nonsense, that's not how index() works. Fixed code further down.

yorickdowne commented Dec 3, 2016

Okay I have something that works.

Edit: That code was nonsense, that's not how index() works. Fixed code further down.

@genio

This comment has been minimized.

Show comment
Hide comment
@genio

genio Dec 3, 2016

irc.perl.org #p5p or the Perl 5 Porters Mailing list

I've made the IRC channel aware of the discussion, but it can't hurt to start a mailing list discussion with the group about it.

genio commented Dec 3, 2016

irc.perl.org #p5p or the Perl 5 Porters Mailing list

I've made the IRC channel aware of the discussion, but it can't hurt to start a mailing list discussion with the group about it.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 3, 2016

Thank you @genio! I sent something out to the mailing list.

Thank you @genio! I sent something out to the mailing list.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 3, 2016

Upon discussion in the mailing list, here's the code. This would go into /usr/share/perl/5.18/File/Find.pm .

# These are hard-coded for now, but may move to hint files.
if ($^O eq 'VMS') {
    $Is_VMS = 1;
    $File::Find::dont_use_nlink  = 1;
}
elsif ($^O eq 'MSWin32') {
    $Is_Win32 = 1;
}
elsif ($^O eq 'linux') {
    if (open my $pv, '<', '/proc/version') {
        my $lv = <$pv>;
        if (index($lv,'Microsoft') != -1 || index($lv,'WSL') != -1) {
            $File::Find::dont_use_nlink = 1;
        }
    }
}

yorickdowne commented Dec 3, 2016

Upon discussion in the mailing list, here's the code. This would go into /usr/share/perl/5.18/File/Find.pm .

# These are hard-coded for now, but may move to hint files.
if ($^O eq 'VMS') {
    $Is_VMS = 1;
    $File::Find::dont_use_nlink  = 1;
}
elsif ($^O eq 'MSWin32') {
    $Is_Win32 = 1;
}
elsif ($^O eq 'linux') {
    if (open my $pv, '<', '/proc/version') {
        my $lv = <$pv>;
        if (index($lv,'Microsoft') != -1 || index($lv,'WSL') != -1) {
            $File::Find::dont_use_nlink = 1;
        }
    }
}
@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 4, 2016

@genio, P5P maintainers are a friendly and gracious bunch. I've received help and direction to satisfy the Ubuntu perl maintainers first, then they can submit upstream to P5P.

I've filed a bug for this in the Ubuntu system: https://bugs.launchpad.net/ubuntu/+source/perl/+bug/1647120

@genio, P5P maintainers are a friendly and gracious bunch. I've received help and direction to satisfy the Ubuntu perl maintainers first, then they can submit upstream to P5P.

I've filed a bug for this in the Ubuntu system: https://bugs.launchpad.net/ubuntu/+source/perl/+bug/1647120

@Leont

This comment has been minimized.

Show comment
Hide comment
@Leont

Leont Dec 4, 2016

How is this handled in find(1)? I would expect it to be broken too when not using -noleaf, yet apparently it isn't.

Leont commented Dec 4, 2016

How is this handled in find(1)? I would expect it to be broken too when not using -noleaf, yet apparently it isn't.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 5, 2016

@Leont I'm not quite sure. I tried to go through the util.c in find(1), but I'm a bit lost as to how the auto-detection works. I know oldfind.c reads the entries, checks, and if stat doesn't match up, set's -noleaf automatically.
How current find handled that I didn't fully grasp. I'm assuming it has a similar detection mechanism somewhere.

@Leont I'm not quite sure. I tried to go through the util.c in find(1), but I'm a bit lost as to how the auto-detection works. I know oldfind.c reads the entries, checks, and if stat doesn't match up, set's -noleaf automatically.
How current find handled that I didn't fully grasp. I'm assuming it has a similar detection mechanism somewhere.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 5, 2016

James E. Keenan has a patch for this that also tidies up how dont_use_nlink is set. https://rt.perl.org/Public/Bug/Display.html?id=130258

James E. Keenan has a patch for this that also tidies up how dont_use_nlink is set. https://rt.perl.org/Public/Bug/Display.html?id=130258

@Leont

This comment has been minimized.

Show comment
Hide comment
@Leont

Leont Dec 5, 2016

How current find handled that I didn't fully grasp. I'm assuming it has a similar detection mechanism somewhere.

File::Find also tries to detect this, in particular the nlink logic is disabled if is sees a directory with less than 2 links, but apparently it isn't triggered in this particular case.

Leont commented Dec 5, 2016

How current find handled that I didn't fully grasp. I'm assuming it has a similar detection mechanism somewhere.

File::Find also tries to detect this, in particular the nlink logic is disabled if is sees a directory with less than 2 links, but apparently it isn't triggered in this particular case.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 5, 2016

Confirmed that this issue is also present in WSL Insider build 14965 with Ubuntu 16.04 LTS and perl v5.22.1. As we expected it would be, just crossing t's and making sure.

Confirmed that this issue is also present in WSL Insider build 14965 with Ubuntu 16.04 LTS and perl v5.22.1. As we expected it would be, just crossing t's and making sure.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 5, 2016

#910 might mean this gets fixed in WSL. It's not fixed there now, mind you, as far as I can tell. But if WSL learns how to handle the nlink hack, efforts to change File::Find should probably stop.

#910 might mean this gets fixed in WSL. It's not fixed there now, mind you, as far as I can tell. But if WSL learns how to handle the nlink hack, efforts to change File::Find should probably stop.

@Leont

This comment has been minimized.

Show comment
Hide comment
@Leont

Leont Dec 6, 2016

File::Find also tries to detect this, in particular the nlink logic is disabled if is sees a directory with less than 2 links, but apparently it isn't triggered in this particular case.

Can someone tell me what the link count of such a directory on WSL is? (stat $dirname should tell you), along with the number of subdirectories.

Is the former the constant for different directories? Is the difference constant?

Leont commented Dec 6, 2016

File::Find also tries to detect this, in particular the nlink logic is disabled if is sees a directory with less than 2 links, but apparently it isn't triggered in this particular case.

Can someone tell me what the link count of such a directory on WSL is? (stat $dirname should tell you), along with the number of subdirectories.

Is the former the constant for different directories? Is the difference constant?

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 6, 2016

tbehrens@DESKTOP-J1SS16J:~$ tree Test
Test
└── a
    ├── b
    │   ├── f1.txt
    │   └── f2.txt
    └── c

3 directories, 2 files
tbehrens@DESKTOP-J1SS16J:~$ stat Test
  File: 'Test'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 1125899906978852  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880282000 -0500
Modify: 2016-12-06 11:08:09.880575400 -0500
Change: 2016-12-06 11:08:09.880575400 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a
  File: 'Test/a'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 1125899906978859  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880509600 -0500
Modify: 2016-12-06 11:08:12.544527000 -0500
Change: 2016-12-06 11:08:12.544527000 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a/b
  File: 'Test/a/b'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 5910974511060019  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880637000 -0500
Modify: 2016-12-06 11:08:22.798697400 -0500
Change: 2016-12-06 11:08:22.798697400 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a/c
  File: 'Test/a/c'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 3096224743953465  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:12.544411100 -0500
Modify: 2016-12-06 11:08:12.544411100 -0500
Change: 2016-12-06 11:08:12.544411100 -0500
 Birth: -

And Ryan's way:

tbehrens@DESKTOP-J1SS16J:~/Test$ find . | xargs -n 1 stat | grep "File\\|Links"
  File: '.'
Device: 11h/17d Inode: 1125899906978852  Links: 0
  File: './a'
Device: 11h/17d Inode: 1125899906978859  Links: 0
  File: './a/b'
Device: 11h/17d Inode: 5910974511060019  Links: 0
  File: './a/b/f1.txt'
Device: 11h/17d Inode: 1407374883689533  Links: 1
  File: './a/b/f2.txt'
Device: 11h/17d Inode: 1125899906978878  Links: 1
  File: './a/c'
Device: 11h/17d Inode: 3096224743953465  Links: 0

yorickdowne commented Dec 6, 2016

tbehrens@DESKTOP-J1SS16J:~$ tree Test
Test
└── a
    ├── b
    │   ├── f1.txt
    │   └── f2.txt
    └── c

3 directories, 2 files
tbehrens@DESKTOP-J1SS16J:~$ stat Test
  File: 'Test'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 1125899906978852  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880282000 -0500
Modify: 2016-12-06 11:08:09.880575400 -0500
Change: 2016-12-06 11:08:09.880575400 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a
  File: 'Test/a'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 1125899906978859  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880509600 -0500
Modify: 2016-12-06 11:08:12.544527000 -0500
Change: 2016-12-06 11:08:12.544527000 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a/b
  File: 'Test/a/b'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 5910974511060019  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:09.880637000 -0500
Modify: 2016-12-06 11:08:22.798697400 -0500
Change: 2016-12-06 11:08:22.798697400 -0500
 Birth: -
tbehrens@DESKTOP-J1SS16J:~$ stat Test/a/c
  File: 'Test/a/c'
  Size: 0               Blocks: 0          IO Block: 512    directory
Device: 11h/17d Inode: 3096224743953465  Links: 0
Access: (0777/drwxrwxrwx)  Uid: ( 1000/tbehrens)   Gid: ( 1000/tbehrens)
Access: 2016-12-06 11:08:12.544411100 -0500
Modify: 2016-12-06 11:08:12.544411100 -0500
Change: 2016-12-06 11:08:12.544411100 -0500
 Birth: -

And Ryan's way:

tbehrens@DESKTOP-J1SS16J:~/Test$ find . | xargs -n 1 stat | grep "File\\|Links"
  File: '.'
Device: 11h/17d Inode: 1125899906978852  Links: 0
  File: './a'
Device: 11h/17d Inode: 1125899906978859  Links: 0
  File: './a/b'
Device: 11h/17d Inode: 5910974511060019  Links: 0
  File: './a/b/f1.txt'
Device: 11h/17d Inode: 1407374883689533  Links: 1
  File: './a/b/f2.txt'
Device: 11h/17d Inode: 1125899906978878  Links: 1
  File: './a/c'
Device: 11h/17d Inode: 3096224743953465  Links: 0
@ryanerwin

This comment has been minimized.

Show comment
Hide comment
@ryanerwin

ryanerwin Dec 6, 2016

@Leont,

Here you go:

mkdir -p test1/{alpha}; touch test1/{a}; mkdir -p test2/{alpha,beta}; touch test2/{b,c}; mkdir -p test3/{alpha,beta,gamma}; touch test3/{a,b,c}

➜  % tree                                                                                                                                          0:15.
├── test1
│   ├── {a}
│   └── {alpha}
├── test2
│   ├── alpha
│   ├── b
│   ├── beta
│   └── c
└── test3
    ├── a
    ├── alpha
    ├── b
    ├── beta
    ├── c
    └── gamma

9 directories, 6 files

➜  % find . | xargs -n 1 stat | grep "File\\|Links"                                                                                                0:15
  File: ‘.’
Device: 1h/1d   Inode: 16607023625939185  Links: 2
  File: ‘./test1’
Device: 1h/1d   Inode: 27021597764469325  Links: 2
  File: ‘./test1/{alpha}’
Device: 1h/1d   Inode: 9288674231855754  Links: 2
  File: ‘./test1/{a}’
Device: 1h/1d   Inode: 13792273859226271  Links: 1
  File: ‘./test2’
Device: 1h/1d   Inode: 10696049115409146  Links: 2
  File: ‘./test2/alpha’
Device: 1h/1d   Inode: 16044073672911611  Links: 2
  File: ‘./test2/b’
Device: 1h/1d   Inode: 7599824371592357  Links: 1
  File: ‘./test2/beta’
Device: 1h/1d   Inode: 4503599627775068  Links: 2
  File: ‘./test2/c’
Device: 1h/1d   Inode: 14636698789358796  Links: 1
  File: ‘./test3’
Device: 1h/1d   Inode: 5629499534617819  Links: 2
  File: ‘./test3/a’
Device: 1h/1d   Inode: 8162774325014664  Links: 1
  File: ‘./test3/alpha’
Device: 1h/1d   Inode: 6473924464749840  Links: 2
  File: ‘./test3/b’
Device: 1h/1d   Inode: 8444249301725472  Links: 1
  File: ‘./test3/beta’
Device: 1h/1d   Inode: 4503599627775337  Links: 2
  File: ‘./test3/c’
Device: 1h/1d   Inode: 5066549581197810  Links: 1
  File: ‘./test3/gamma’
Device: 1h/1d   Inode: 9288674231856492  Links: 2

ryanerwin commented Dec 6, 2016

@Leont,

Here you go:

mkdir -p test1/{alpha}; touch test1/{a}; mkdir -p test2/{alpha,beta}; touch test2/{b,c}; mkdir -p test3/{alpha,beta,gamma}; touch test3/{a,b,c}

➜  % tree                                                                                                                                          0:15.
├── test1
│   ├── {a}
│   └── {alpha}
├── test2
│   ├── alpha
│   ├── b
│   ├── beta
│   └── c
└── test3
    ├── a
    ├── alpha
    ├── b
    ├── beta
    ├── c
    └── gamma

9 directories, 6 files

➜  % find . | xargs -n 1 stat | grep "File\\|Links"                                                                                                0:15
  File: ‘.’
Device: 1h/1d   Inode: 16607023625939185  Links: 2
  File: ‘./test1’
Device: 1h/1d   Inode: 27021597764469325  Links: 2
  File: ‘./test1/{alpha}’
Device: 1h/1d   Inode: 9288674231855754  Links: 2
  File: ‘./test1/{a}’
Device: 1h/1d   Inode: 13792273859226271  Links: 1
  File: ‘./test2’
Device: 1h/1d   Inode: 10696049115409146  Links: 2
  File: ‘./test2/alpha’
Device: 1h/1d   Inode: 16044073672911611  Links: 2
  File: ‘./test2/b’
Device: 1h/1d   Inode: 7599824371592357  Links: 1
  File: ‘./test2/beta’
Device: 1h/1d   Inode: 4503599627775068  Links: 2
  File: ‘./test2/c’
Device: 1h/1d   Inode: 14636698789358796  Links: 1
  File: ‘./test3’
Device: 1h/1d   Inode: 5629499534617819  Links: 2
  File: ‘./test3/a’
Device: 1h/1d   Inode: 8162774325014664  Links: 1
  File: ‘./test3/alpha’
Device: 1h/1d   Inode: 6473924464749840  Links: 2
  File: ‘./test3/b’
Device: 1h/1d   Inode: 8444249301725472  Links: 1
  File: ‘./test3/beta’
Device: 1h/1d   Inode: 4503599627775337  Links: 2
  File: ‘./test3/c’
Device: 1h/1d   Inode: 5066549581197810  Links: 1
  File: ‘./test3/gamma’
Device: 1h/1d   Inode: 9288674231856492  Links: 2
@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 6, 2016

Ryan, which build is this? I'm a slightly older one, 14965 (because UUP work that started Friday the 2nd, all I had was ISO), and I'm getting 0 links for directories.

If this is fixed that'd be great news. Can you try:

perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'

And see whether it works recursively in your build? Assuming you didn't change don't_use_nlink that is, if you did, that will of course work.

Ryan, which build is this? I'm a slightly older one, 14965 (because UUP work that started Friday the 2nd, all I had was ISO), and I'm getting 0 links for directories.

If this is fixed that'd be great news. Can you try:

perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'

And see whether it works recursively in your build? Assuming you didn't change don't_use_nlink that is, if you did, that will of course work.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 6, 2016

I think I have to walk my statement back. Upon further testing, recursive File::Find appears to work in my build, and I can install modules from cpan. I fiddled with File::Find previously, but best as I can tell, I have removed those changes and dont_use_nlink is not being set manually.

It'd be good to have that confirmed.

I think I have to walk my statement back. Upon further testing, recursive File::Find appears to work in my build, and I can install modules from cpan. I fiddled with File::Find previously, but best as I can tell, I have removed those changes and dont_use_nlink is not being set manually.

It'd be good to have that confirmed.

@Leont

This comment has been minimized.

Show comment
Hide comment
@Leont

Leont Dec 6, 2016

It'd be good to have that confirmed.

A link count of 0 on the directory should disable the nlink optimization AFAICT. The whole optimization is assuming the linkcount == $subdirs + 2, and that evidently can't be the case if $linkcount < 2 on a directory.

Having $linkcount == 2 no matter the number of subdirectories is exactly what would explain the previously observed results.

Leont commented Dec 6, 2016

It'd be good to have that confirmed.

A link count of 0 on the directory should disable the nlink optimization AFAICT. The whole optimization is assuming the linkcount == $subdirs + 2, and that evidently can't be the case if $linkcount < 2 on a directory.

Having $linkcount == 2 no matter the number of subdirectories is exactly what would explain the previously observed results.

@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 6, 2016

Right. So that'd mean my testing yesterday evening was sloppy, and MS did fix this in an insider build by forcing the link count to 0.

Right. So that'd mean my testing yesterday evening was sloppy, and MS did fix this in an insider build by forcing the link count to 0.

@ryanerwin

This comment has been minimized.

Show comment
Hide comment
@ryanerwin

ryanerwin Dec 6, 2016

I've been testing on the default "Anniversary Edition" version of WSL, and switching between the default Ubuntu image and a CentOS 7 image.

When Config.pm: dont_use_nlink => ''

➜  % perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'                                                                         0:47.
./test1
./test2
./test3

When Config.pm dont_use_nlink => 'defined'

➜  % perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'                                                                         0:50.
./test1
./test1/{a}
./test1/{alpha}
./test2
./test2/b
./test2/c
./test2/alpha
./test2/beta
./test3
./test3/a
./test3/b
./test3/c
./test3/alpha
./test3/beta
./test3/gamma

I've been testing on the default "Anniversary Edition" version of WSL, and switching between the default Ubuntu image and a CentOS 7 image.

When Config.pm: dont_use_nlink => ''

➜  % perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'                                                                         0:47.
./test1
./test2
./test3

When Config.pm dont_use_nlink => 'defined'

➜  % perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'                                                                         0:50.
./test1
./test1/{a}
./test1/{alpha}
./test2
./test2/b
./test2/c
./test2/alpha
./test2/beta
./test3
./test3/a
./test3/b
./test3/c
./test3/alpha
./test3/beta
./test3/gamma
@yorickdowne

This comment has been minimized.

Show comment
Hide comment
@yorickdowne

yorickdowne Dec 6, 2016

Thanks @ryanerwin ! I think that's conclusive. In my system, dont_use_nlink => undef and as best I can tell I've removed the code I had previously added to File::Find, and I get:

tbehrens@DESKTOP-J1SS16J:~/Test$ perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'
.
./a
./a/b
./a/b/f1.txt
./a/b/f2.txt
./a/c

That'd mean MS has fixed it in Insider build, and therefore in Creator's Update, and my whole push towards changes in File::Find was misguided.

Thanks @ryanerwin ! I think that's conclusive. In my system, dont_use_nlink => undef and as best I can tell I've removed the code I had previously added to File::Find, and I get:

tbehrens@DESKTOP-J1SS16J:~/Test$ perl -e 'use File::Find;find(sub {print $File::Find::name,$/;},".");'
.
./a
./a/b
./a/b/f1.txt
./a/b/f2.txt
./a/c

That'd mean MS has fixed it in Insider build, and therefore in Creator's Update, and my whole push towards changes in File::Find was misguided.

@jackchammons

This comment has been minimized.

Show comment
Hide comment
@jackchammons

jackchammons Dec 14, 2016

Member

@ALL As of build number 14986 I am no longer able to reproduce the original error that spawned this thread. Based on @yorickdowne 's comment above it looks like this issue has been fixed.

Feel free to reopen if I am closing prematurely.

Member

jackchammons commented Dec 14, 2016

@ALL As of build number 14986 I am no longer able to reproduce the original error that spawned this thread. Based on @yorickdowne 's comment above it looks like this issue has been fixed.

Feel free to reopen if I am closing prematurely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment