Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter: fix directory filter globs for / and ** in {} #3250

Open
faxotherapy opened this issue Jun 8, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@faxotherapy
Copy link

commented Jun 8, 2019

What is the problem you are having with rclone?

In order to replicate the issue, I copied my .ssh folder tree while only keeping two files.

$ mkdir -p ~/ssh/vault/keys
$ cp ~/.ssh/config ~/ssh/
$ cp ~/.ssh/vault/keys/domus_rsa.pub ~/ssh/vault/keys/
$ tree ~/ssh
/Users/N/ssh
├── config
└── vault
    └── keys
        └── domus_rsa.pub

2 directories, 2 files

I create filter_test.txt:

+ /\.gitconfig
+ /\.vimrc
+ /\.zshrc
+ /\.tmux.conf

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put in practice filter_test.txt:

$ rclone --filter-from /usr/local/etc/rclone/filter_test.txt sync ~ Safe:/TEST/

I visit folder TEST content:

$ rclone ls Safe:TEST
      387 .gitconfig
     4906 .tmux.conf
     1930 .vimrc
    32405 .zshrc
     2105 ssh/config
      736 ssh/vault/keys/domus_rsa.pub

Now, I create filter_test_2.txt:

- /**/.DS_Store

+ /ssh/{config,vault/keys/**}

- *

I put filter_test_2.txt in practice:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/TEST
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And, filter works as expected! But…

Let's do something else: applying filter_test_2.txt to my actual backup folder:

rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/MBP

Bingo! The filter is ignored and all my MBP directory is listed!

So, I don't understand why applying filter_test_2.txt to folder TEST produces expected results and not to MBP folder. I'm completely bewildered. What's inside MBP folder to cause the filter to break? It's a complete mystery!

I create filter_test_3.txt:

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put filter_test_3.txt in practice with my real life backup folder:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_3.txt Safe:/MBP
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And it works! So, filter_test_2.txt does not work with my MBP folder.

What is your rclone version (output from rclone version)

rclone v1.47.0
- os/arch: darwin/amd64
- go version: go1.12.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS Mojave 10.14.5

Which cloud storage system are you using? (eg Google Drive)

SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/MBP

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This issue has already been extensively discussed in this rclone forum.

Thank you.

@ncw

This comment has been minimized.

Copy link
Owner

commented Jun 8, 2019

I've managed to reproduce that.

 rclone lsf -R --include '/a/{b,c/**}' .  --dump filters
--- start filters ---
--- File filter rules ---
+ ^a/(b|c/.*)$
- ^.*$
--- Directory filter rules ---
+ ^.*$     <---- *** THIS IS WRONG ***!
+ ^a/(b|c/.*)$
- ^.*$
--- end filters ---

There is a directory include filter which shouldn't be there.

Actually reading the code I see this

rclone/fs/filter/glob.go

Lines 124 to 125 in 454dfd3

// Can't deal with / or ** in {}
tooHardRe = regexp.MustCompile(`{[^{}]*(\*\*|/)[^{}]*}`)

rclone/fs/filter/glob.go

Lines 135 to 140 in 454dfd3

func globToDirGlobs(glob string) (out []string) {
if tooHardRe.MatchString(glob) {
// Can't figure this one out so return any directory might match
out = append(out, "/**")
return out
}

So when I wrote the code I punted this difficult case ;-)

The consequences are that you'll see every directory in the search path but you shouldn't see all the files.

That should produce a warning I suppose, or I should fix it...

@ncw ncw changed the title Filter Simply Ignored for Some Specific Target Backup Folders filter: fix directory filter globs for / and ** in {} Jun 8, 2019

@ncw ncw added the bug label Jun 8, 2019

@ncw ncw added this to the v1.49 milestone Jun 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.