filter: ability to use negative patterns

This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix restic#233

changelog/unreleased/issue-233

@@ -0,0 +1,31 @@
+Enhancement: Add negative patterns for include/exclude
+
+If a pattern is suffixed by an exclamation mark and match a file that
+was previously matched by a regular pattern, the match is cancelled.
+Notably, this can be used with `--exclude-file` to cancel the
+exclusion of some files.
+
+It works similarly to `gitignore`, with the same limitation: once a
+directory is excluded, it is not possible to include files inside the
+directory.
+
+Example of use (as an exclude pattern for backup):
+
+    $HOME/**/*
+    !$HOME/Documents
+    !$HOME/code
+    !$HOME/.emacs.d
+    !$HOME/games
+    # [...]
+    node_modules
+    *~
+    *.o
+    *.lo
+    *.pyc
+    # [...]
+    $HOME/code/linux/*
+    !$HOME/code/linux/.git
+    # [...]
+
+https://github.com/restic/restic/issues/233
+https://github.com/restic/restic/pull/2311

doc/040_backup.rst

@@ -195,6 +195,28 @@ sub-directories: The pattern ``foo/**/bar`` matches:
  * ``/foo/bar/file``
  * ``/tmp/foo/bar``
 
+If a pattern is suffixed by an exclamation mark and match a file that
+was previously matched by a regular pattern, the match is cancelled.
+It works similarly to ``gitignore``, with the same limitation: once a
+directory is excluded, it is not possible to include files inside the
+directory. Here is a complete example to backup a selection of
+directories inside the home directory. It works by excluding any
+directory, then selectively add back some of them.
+
+::
+
+    $HOME/**/*
+    !$HOME/Documents
+    !$HOME/code
+    !$HOME/.emacs.d
+    !$HOME/games
+    # [...]
+    node_modules
+    *~
+    *.o
+    *.lo
+    *.pyc
+
 By specifying the option ``--one-file-system`` you can instruct restic
 to only backup files from the file systems the initially specified files
 or directories reside on. For example, calling restic like this won't

internal/filter/filter.go

@@ -159,13 +159,20 @@ func match(patterns, strs []string) (matched bool, err error) {
 	return false, nil
 }
 
-// List returns true if str matches one of the patterns. Empty patterns are
-// ignored.
+// List returns true if str matches one of the patterns. Empty
+// patterns are ignored. Patterns prefixed by "!" are negated: any
+// matching file excluded by a previous pattern will become included
+// again.
 func List(patterns []string, str string) (matched bool, childMayMatch bool, err error) {
 	for _, pat := range patterns {
+		var negate bool
 		if pat == "" {
 			continue
 		}
+		if pat[0] == '!' {
+			negate = true
+			pat = pat[1:]
+		}
 
 		m, err := Match(pat, str)
 		if err != nil {
@@ -177,12 +184,8 @@ func List(patterns []string, str string) (matched bool, childMayMatch bool, err
 			return false, false, err
 		}
 
-		matched = matched || m
-		childMayMatch = childMayMatch || c
-
-		if matched && childMayMatch {
-			return true, true, nil
-		}
+		matched = matched && !m || m && !negate
+		childMayMatch = childMayMatch && !(m && c) || c && !negate
 	}
 
 	return matched, childMayMatch, nil

internal/filter/filter_test.go

@@ -257,7 +257,20 @@ var filterListTests = []struct {
 	{[]string{"/*/*/bar/test.*"}, "/foo/bar/bar", false, true},
 	{[]string{"/*/*/bar/test.*", "*.go"}, "/foo/bar/test.go", true, true},
 	{[]string{"", "*.c"}, "/foo/bar/test.go", false, true},
+	{[]string{"!**", "*.go"}, "/foo/bar/test.go", true, true},
+	{[]string{"!**", "*.c"}, "/foo/bar/test.go", false, true},
+	{[]string{"/foo/*/test.*", "!*.c"}, "/foo/bar/test.c", false, false},
+	{[]string{"/foo/*/test.*", "!*.c"}, "/foo/bar/test.go", true, true},
+	{[]string{"/foo/*/*", "!test.*", "*.c"}, "/foo/bar/test.go", false, true},
+	{[]string{"/foo/*/*", "!test.*", "*.c"}, "/foo/bar/test.c", true, true},
+	{[]string{"/foo/*/*", "!test.*", "*.c"}, "/foo/bar/file.go", true, true},
+	{[]string{"/**/*", "!/foo", "/foo/*", "!/foo/bar"}, "/foo/other/test.go", true, true},
+	{[]string{"/**/*", "!/foo", "/foo/*", "!/foo/bar"}, "/foo/bar", false, false},
+	{[]string{"/**/*", "!/foo", "/foo/*", "!/foo/bar"}, "/foo/bar/test.go", false, false},
+	{[]string{"/**/*", "!/foo", "/foo/*", "!/foo/bar"}, "/foo/bar/test.go/child", false, false},
+	{[]string{"/**/*", "!/foo", "/foo/*", "!/foo/bar", "/foo/bar/test*"}, "/foo/bar/test.go/child", true, true},
 	{[]string{"/foo/bar/*"}, "/foo", false, true},
+	{[]string{"/foo/bar/*", "!/foo/bar/[a-m]*"}, "/foo", false, true},
 	{[]string{"/foo/**/test.c"}, "/foo/bar/foo/bar/test.c", true, true},
 	{[]string{"/foo/*/test.c"}, "/foo/bar/foo/bar/test.c", false, false},
 }