File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem #16759
Comments
From @jikamensCreated by @jikamensThis is a bug report for perl from jik@kamens.us, ----------------------------------------------------------------- This script has worked fine in the past, but it is no longer iterating I looked at the code in Find.pm and I see that it automatically avoids Unfortunately, it appears that directories on the NAS return 2 as the $ cd ~/ogg Compare with the local filesystem in /tmp: $ cd /tmp I don't think this was caused by a change to the code in File::Find, So while the problem may not have started happening due to a change in Perl Info |
From @jkeenanOn Tue, 20 Nov 2018 22:51:54 GMT, jik@kamens.us wrote:
For those of us who, like myself, are unfamiliar with your terminology ... are these the "CIFS" and "Synology NAS" you are referring to? https://en.wikipedia.org/wiki/Server_Message_Block
-- |
|
The RT System itself - Status changed from 'new' to 'open' |
From @TuxOn Tue, 20 Nov 2018 18:32:32 -0800, "James E Keenan via RT"
CIFS ~= Samba : https://www.varonis.com/blog/cifs-vs-smb/ HP-UX also uses CIFS to refer to cross-system mounts à la Windows, but Here, it is relative important what DSM (the Synology NAS OS) version See also README.synology in the perl root (which might need some I know that the OP uses perl on a debian system to a SMB mount on the On my DS218, I have 2 perl versions available: Linux 4.4.59+ [DSM 6.2-23824-1] DS218+ Celeron(R) CPU J3355 @ 2.00GHz/2001(2) x86_64 1828 Mb /opt/bin/perl /usr/local/bin/perl
I can confirm that, but have to add that this has nothing to do with
/NAS/Tmp > stat foo
/NAS/Tmp > stat foo/bar
/tmp 509 > stat foo
/tmp 510 > stat foo/bar
Now on the NAS itself: /volume1/NAS/Tmp > stat foo /volume1/NAS/Tmp > stat foo/bar This all has to do with the FS implementation of DSM and the -- |
From @jikamensI have a DS415+ NAS running DSM 6.2.1-23824 Update 1, the current release. jik |
From @tonycozOn Tue, 20 Nov 2018 14:51:54 -0800, jik@kamens.us wrote:
I wonder if this directory size of zero can be used as a heuristic for distrusting st_nlink.
Unfortunately I don't see a simple fix for File::Find itself. As a workaround you could do: local $File::Find::dont_use_link = 1; before calling File::Find functions. In terms of fixing File::Find itself, all I can see is distrusting nlink on a given st_dev until we've found a directory with st_nlink > 2. Tony |
From Stromeko@nexgo.de"Tony Cook via RT" writes:
I don't see the relation. Many non-UN*X file systems these days don't If directory links are not implemented, the reported number of links
Well, the first directory you find where nlink==2 and there is at least Regards, Factory and User Sound Singles for Waldorf Blofeld: |
From @LeontOn Mon, Nov 26, 2018 at 5:47 AM Tony Cook via RT
Quite frankly, this optimization has given us sufficient headache over Leon |
From @tonycozOn Mon, 26 Nov 2018 11:01:52 -0800, LeonT wrote:
Like the attached. Tony |
From @tonycoz0001-perl-133673-disable-the-File-Find-nlink-optimization.patchFrom e3ad92c0a5ef0a707b5cb4e0d98fda313f3442c7 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Mon, 10 Dec 2018 15:38:54 +1100
Subject: (perl #133673) disable the File::Find nlink optimization by default
This will also fix 128894 and 126144.
---
ext/File-Find/lib/File/Find.pm | 40 +++++++++-------------------------------
1 file changed, 9 insertions(+), 31 deletions(-)
diff --git a/ext/File-Find/lib/File/Find.pm b/ext/File-Find/lib/File/Find.pm
index 03dac9fbda..c052ceebb3 100644
--- a/ext/File-Find/lib/File/Find.pm
+++ b/ext/File-Find/lib/File/Find.pm
@@ -3,7 +3,7 @@ use 5.006;
use strict;
use warnings;
use warnings::register;
-our $VERSION = '1.34';
+our $VERSION = '1.35';
require Exporter;
require Cwd;
@@ -770,31 +770,11 @@ sub finddepth {
$File::Find::skip_pattern = qr/^\.{1,2}\z/;
$File::Find::untaint_pattern = qr|^([-+@\w./]+)$|;
-# These are hard-coded for now, but may move to hint files.
-if ($^O eq 'VMS') {
- $Is_VMS = 1;
- $File::Find::dont_use_nlink = 1;
-}
-elsif ($^O eq 'MSWin32') {
- $Is_Win32 = 1;
-}
-
# this _should_ work properly on all platforms
# where File::Find can be expected to work
$File::Find::current_dir = File::Spec->curdir || '.';
-$File::Find::dont_use_nlink = 1
- if $^O eq 'os2' || $^O eq 'dos' || $^O eq 'amigaos' || $Is_Win32 ||
- $^O eq 'interix' || $^O eq 'cygwin' || $^O eq 'qnx' || $^O eq 'nto';
-
-# Set dont_use_nlink in your hint file if your system's stat doesn't
-# report the number of links in a directory as an indication
-# of the number of files.
-# See e.g. hints/haiku.sh for Haiku.
-unless ($File::Find::dont_use_nlink) {
- require Config;
- $File::Find::dont_use_nlink = 1 if ($Config::Config{'dont_use_nlink'});
-}
+$File::Find::dont_use_nlink = 1;
# We need a function that checks if a scalar is tainted. Either use the
# Scalar::Util module's tainted() function or our (slower) pure Perl
@@ -1106,17 +1086,15 @@ warnings.
=item $dont_use_nlink
-You can set the variable C<$File::Find::dont_use_nlink> to 1 if you want to
-force File::Find to always stat directories. This was used for file systems
-that do not have an C<nlink> count matching the number of sub-directories.
-Examples are ISO-9660 (CD-ROM), AFS, HPFS (OS/2 file system), FAT (DOS file
-system) and a couple of others.
+You can set the variable C<$File::Find::dont_use_nlink> to 0 if you
+are sure the filesystem you are scanning reflects the number of
+subdirectories in the parent directory's C<nlink> count.
-You shouldn't need to set this variable, since File::Find should now detect
-such file systems on-the-fly and switch itself to using stat. This works even
-for parts of your file system, like a mounted CD-ROM.
+If you do set C<$File::Find::dont_use_nlink> to 0, you may notice an
+improvement in speed at the risk of not recursing into subdirectories
+if a filesystem doesn't populate C<nlink> as expected.
-If you do set C<$File::Find::dont_use_nlink> to 1, you will notice slow-downs.
+C<$File::Find::dont_use_nlink> now defaults to 1 on all platforms.
=item symlinks
--
2.11.0
|
From @sevanOn 10/12/2018 04:39, Tony Cook via RT wrote:
Does that not break functionality on current environments like QNX, Sevan |
From @tonycozOn Mon, Dec 10, 2018 at 05:04:44AM +0000, Sevan Janiyan wrote:
-$File::Find::dont_use_nlink = 1 is replaced: +$File::Find::dont_use_nlink = 1; further down. So there's no change on Cygwin, QNX, Haiku, etc. Tony |
From @tonycozOn Sun, 09 Dec 2018 20:39:43 -0800, tonyc wrote:
Applied as f75dfdb. Tony |
|
@tonycoz - Status changed from 'open' to 'pending release' |
From @khwilliamsonThank you for filing this report. You have helped make Perl better. With the release today of Perl 5.30.0, this and 160 other issues have been Perl 5.30.0 may be downloaded via: If you find that the problem persists, feel free to reopen this ticket. |
|
@khwilliamson - Status changed from 'pending release' to 'resolved' |
File::Find would not descend into subdirs on smbfs, refer to Perl/perl5#16759. Solution: set $File::Find::dont_use_nlink = 1, as suggested by the above link. Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
* Vagrantfile: fix reading M0_VM_SHARE_TYPE env var * Vagrantfile: add SMB-type shared dir support * Vagrantfile: add vmware_desktop provider * xcode/proto: fix directories traversing on smbfs File::Find would not descend into subdirs on smbfs, refer to Perl/perl5#16759. Solution: set $File::Find::dont_use_nlink = 1, as suggested by the above link. * Vagrantfile: fix disks creation for SSUs on vmware_desktop * Vagrantfile: fix NetworkManager startup issue on vmware_desktop Fixed by avoiding creating hosts-only network interfaces (which are not needed on VMware VMs anyway). Why it caused problems with NetworkManager for VMs which are started after the 1st one (for the 1st VM did not cause any problems) - still not clear. * Vagrantfile: allow to provide SMB credentials via env vars Use SMB_USERNAME and SMB_PASSWORD env variables to provide SMB credentials. * ansible: fix jumphost ssh pub key setup On Windows host cmu's ssh pub key is not set up for other nodes. As result, the password-less access was not working from cmu to these nodes. During the provisioning ansible printed: ``` TASK [ssh-target : authorize SSH jumphost pub key on client1 host] ************* fatal: [client1]: FAILED! => {"msg": "'ansible_user' is undefined"} ...ignoring msg: 'ansible_user' is undefined ``` It's not quite clearly why the 'ansible_user' is undefined for this task, because for the previous task the same 'ansible_user' passed OK. Note: in case of Windows host, ansible is running in the guest OS. The problem was not seen on macOS host nor on Linux host. Solution: make two independent tasks for root and 'ansible_user' instead of doing it in one task with the `with_items` directive - this way it works, as appeared. Not quite clear why. * Vagrantfile: fix vmware_desktop provider for macOS Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com> Reviewed-by: Dima Chumak <dmitriy.chumak@seagate.com>
Migrated from rt.perl.org#133673 (status was 'resolved')
Searchable as RT133673$
The text was updated successfully, but these errors were encountered: