Make the parallel walker visit untraversable directories before failing #1365

jakubadamw · 2019-09-03T22:03:18Z

This PR fixes an inconsistency between the serial and the parallel
directory walkers around visiting a directory for which the user holds
insufficient permissions to descend into.

The serial walker does produce a successful entry for a directory that
it cannot descend into due to insufficient permissions. However, before
this change that has not been the case for the parallel walker, which
would produce an Err item not only when descending into a directory
that it cannot read from but also for the directory entry itself.

This change brings the behaviour of the parallel variant in line with
that of the serial one.

Fixes #1346.

This commit fixes an inconsistency between the serial and the parallel directory walkers around visiting a directory for which the user holds insufficient permissions to descend into. The serial walker does produce a successful entry for a directory that it cannot descend into due to insufficient permissions. However, before this change that has not been the case for the parallel walker, which would produce an `Err` item not only when descending into a directory that it cannot read from but also for the directory entry itself. This change brings the behaviour of the parallel variant in line with that of the serial one.

jakubadamw · 2019-09-04T12:09:24Z

I should probably note that the new test will always fail when running it as a root. Is that a problem?

BurntSushi

Thanks for looking into this!

ignore/src/walk.rs

BurntSushi · 2019-09-04T12:22:09Z

ignore/src/walk.rs

+            let builder = WalkBuilder::new(td.path());
+            assert_paths(td.path(), &builder, &["a"]);
+        }
+    }


Hmm, yeah, tests should not fail depending on what user ran them. Maybe it would be good to check if the file is readable after chmodding it. If it is, then skip the test.

With that said, it would be nice if we could avoid bringing in libc and unsafe for this test. Is there some other way to test this? Nothing is coming to mind... Maybe using a pre-existing file that a normal user traditionally doesn't have access to? (And skipping the test if the file doesn't exist or is readable.)

Please also make sure that lines are wrapped to 79 columns, inclusive.

@BurntSushi, I added the proposed check, though, of course, it'd be better if the harness allowed marking a test as skipped at runtime.

With regard to avoiding adding a dev-dependency on libc I don't have a good idea either. But now I made it a Linux-only dependency. Perhaps that's good enough?

If using libc is really an issue (even though it already is a regular dependency for some other ripgrep crates, though, in truth, not ignore itself), then I'll be happy to change this to try to walk /root and check if it is observing at least one valid entry (i.e. the one for the directory itself).

I probably care more about the extra unsafe to be honest. I'm mostly just trying to make sure things are as a tight as possible. This isn't a huge deal, but I want to try for an alternative if it's feasible.

Just looking at my file system, maybe just trying look for /etc/sudoers.d? It's readable by root, but not by anyone else. So I think the logic might be something like this:

Stat /etc/sudoers.d via std::fs::metadata().

If it errors, skip test.

Read /etc/sudoers.d.

If it succeeds, skip test.

Try to traverse /etc/sudoers.d and run the test normally.

@BurntSushi, alright, sounds like a plan! I pushed the proposed change. Without a doubt that made the test much shorter and simpler. Thanks for the review!

Maybe /root (or more generally ~root) is more ubiquitous than /etc/sudoers.d?

BurntSushi

Okay, I think this LGTM. I'll merge this once I get CI back online. (I tried setting up GitHub Actions over the weekend, but it's completely stuck.)

jakubadamw · 2019-09-04T17:48:55Z

@BurntSushi, there's a button there for cancelling a workflow. Have you tried that? (You probably have and I'm asking a stupid question. 🙂)

BurntSushi · 2019-09-04T17:52:13Z

there's a button there for cancelling a workflow. Have you tried that? (You probably have and I'm asking a stupid question. slightly_smiling_face)

Hah, I wish. But yeah, that button does normally show up. In my case though, Actions seems to be stuck in some sort of initialization state where by that button doesn't show up. The jobs have been running for over 4 days too, which suggests timeout functionality doesn't work either.

I've sent mail to GitHub support but haven't heard back from them.

I've turned Travis and AppVeyor back on. Going to see if I can trigger it on this PR.

treere · 2019-09-21T20:55:41Z

there is a subtle bug and a performance issue. give me some minutes and I will explain.

treere

I think that there is a small bug and a little persormance issue. (one is inside a piece marked as resolved)

treere · 2019-09-21T20:58:55Z

ignore/src/walk.rs

+            // have sufficient read permissions to list the directory.
+            // In that case we still want to provide the closure with a valid
+            // entry before passing the error value.
+            let readdir = work.read_dir();


here there is an unnecessary syscall. I think that a clone is 99% better in performance

I'm not sure I follow your comment here. There is exactly one call to read_dir both before and after this PR.

ignore/src/walk.rs

This commit fixes an inconsistency between the serial and the parallel directory walkers around visiting a directory for which the user holds insufficient permissions to descend into. The serial walker does produce a successful entry for a directory that it cannot descend into due to insufficient permissions. However, before this change that has not been the case for the parallel walker, which would produce an `Err` item not only when descending into a directory that it cannot read from but also for the directory entry itself. This change brings the behaviour of the parallel variant in line with that of the serial one. Fixes #1346, Closes #1365

jakubadamw force-pushed the issue-1346 branch from 329e481 to a08a1b1 Compare September 3, 2019 22:17

BurntSushi requested changes Sep 4, 2019

View reviewed changes

Address review feedback

d285c2b

jakubadamw force-pushed the issue-1346 branch from 4694d52 to d285c2b Compare September 4, 2019 17:37

BurntSushi approved these changes Sep 4, 2019

View reviewed changes

BurntSushi closed this Sep 4, 2019

BurntSushi reopened this Sep 4, 2019

sharkdp mentioned this pull request Sep 17, 2019

Display directories even if we don't have permission to enter sharkdp/fd#437

Closed

treere reviewed Sep 21, 2019

View reviewed changes

BurntSushi added the rollup A PR that has been merged with many others in a rollup. label Feb 17, 2020

BurntSushi closed this in 322fc75 Feb 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the parallel walker visit untraversable directories before failing #1365

Make the parallel walker visit untraversable directories before failing #1365

jakubadamw commented Sep 3, 2019

jakubadamw commented Sep 4, 2019

BurntSushi left a comment

BurntSushi Sep 4, 2019

BurntSushi Sep 4, 2019

jakubadamw Sep 4, 2019 •

edited

Loading

BurntSushi Sep 4, 2019 •

edited

Loading

jakubadamw Sep 4, 2019

tavianator Sep 23, 2019

BurntSushi left a comment

jakubadamw commented Sep 4, 2019

BurntSushi commented Sep 4, 2019

treere commented Sep 21, 2019

treere left a comment •

edited

Loading

treere Sep 21, 2019

BurntSushi Feb 17, 2020

Make the parallel walker visit untraversable directories before failing #1365

Make the parallel walker visit untraversable directories before failing #1365

Conversation

jakubadamw commented Sep 3, 2019

jakubadamw commented Sep 4, 2019

BurntSushi left a comment

Choose a reason for hiding this comment

BurntSushi Sep 4, 2019

Choose a reason for hiding this comment

BurntSushi Sep 4, 2019

Choose a reason for hiding this comment

jakubadamw Sep 4, 2019 • edited Loading

Choose a reason for hiding this comment

BurntSushi Sep 4, 2019 • edited Loading

Choose a reason for hiding this comment

jakubadamw Sep 4, 2019

Choose a reason for hiding this comment

tavianator Sep 23, 2019

Choose a reason for hiding this comment

BurntSushi left a comment

Choose a reason for hiding this comment

jakubadamw commented Sep 4, 2019

BurntSushi commented Sep 4, 2019

treere commented Sep 21, 2019

treere left a comment • edited Loading

Choose a reason for hiding this comment

treere Sep 21, 2019

Choose a reason for hiding this comment

BurntSushi Feb 17, 2020

Choose a reason for hiding this comment

jakubadamw Sep 4, 2019 •

edited

Loading

BurntSushi Sep 4, 2019 •

edited

Loading

treere left a comment •

edited

Loading