filter: fix directory filter globs for / and ** in {}

[faxotherapy]

What is the problem you are having with rclone?

In order to replicate the issue, I copied my .ssh folder tree while only keeping two files.

$ mkdir -p ~/ssh/vault/keys
$ cp ~/.ssh/config ~/ssh/
$ cp ~/.ssh/vault/keys/domus_rsa.pub ~/ssh/vault/keys/
$ tree ~/ssh
/Users/N/ssh
├── config
└── vault
    └── keys
        └── domus_rsa.pub

2 directories, 2 files

I create filter_test.txt:

+ /\.gitconfig
+ /\.vimrc
+ /\.zshrc
+ /\.tmux.conf

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put in practice filter_test.txt:

$ rclone --filter-from /usr/local/etc/rclone/filter_test.txt sync ~ Safe:/TEST/

I visit folder TEST content:

$ rclone ls Safe:TEST
      387 .gitconfig
     4906 .tmux.conf
     1930 .vimrc
    32405 .zshrc
     2105 ssh/config
      736 ssh/vault/keys/domus_rsa.pub

Now, I create filter_test_2.txt:

- /**/.DS_Store

+ /ssh/{config,vault/keys/**}

- *

I put filter_test_2.txt in practice:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/TEST
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And, filter works as expected! But…

Let's do something else: applying filter_test_2.txt to my actual backup folder:

rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/MBP

Bingo! The filter is ignored and all my MBP directory is listed!

So, I don't understand why applying filter_test_2.txt to folder TEST produces expected results and not to MBP folder. I'm completely bewildered. What's inside MBP folder to cause the filter to break? It's a complete mystery!

I create filter_test_3.txt:

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put filter_test_3.txt in practice with my real life backup folder:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_3.txt Safe:/MBP
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And it works! So, filter_test_2.txt does not work with my MBP folder.

What is your rclone version (output from rclone version)

rclone v1.47.0
- os/arch: darwin/amd64
- go version: go1.12.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS Mojave 10.14.5

Which cloud storage system are you using? (eg Google Drive)

SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/MBP

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This issue has already been extensively discussed in this rclone forum.

Thank you.

[ncw]

I've managed to reproduce that.

 rclone lsf -R --include '/a/{b,c/**}' .  --dump filters
--- start filters ---
--- File filter rules ---
+ ^a/(b|c/.*)$
- ^.*$
--- Directory filter rules ---
+ ^.*$     <---- *** THIS IS WRONG ***!
+ ^a/(b|c/.*)$
- ^.*$
--- end filters ---

There is a directory include filter which shouldn't be there.

Actually reading the code I see this

rclone/fs/filter/glob.go

Lines 124 to 125 in 454dfd3

	// Can't deal with / or ** in {}
	tooHardRe = regexp.MustCompile(`{[^{}]*(\*\*|/)[^{}]*}`)

rclone/fs/filter/glob.go

Lines 135 to 140 in 454dfd3

	func globToDirGlobs(glob string) (out []string) {
	if tooHardRe.MatchString(glob) {
	// Can't figure this one out so return any directory might match
	out = append(out, "/**")
	return out
	}

So when I wrote the code I punted this difficult case ;-)

The consequences are that you'll see every directory in the search path but you shouldn't see all the files.

That should produce a warning I suppose, or I should fix it...