Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow mixing include and exclude patterns for backup and restore #233

Open
fd0 opened this issue Jul 18, 2015 · 17 comments · May be fixed by #2311
Open

Allow mixing include and exclude patterns for backup and restore #233

fd0 opened this issue Jul 18, 2015 · 17 comments · May be fixed by #2311

Comments

@fd0
Copy link
Member

@fd0 fd0 commented Jul 18, 2015

This is a followup issue for #226. Up to now it is only possible to specify exclude patterns for backup and restore.

There are uses cases that aren't covered, for example: A user would like to backup her home directory at /home/user, excluding anything from the directory work except C source files.

Are there any other uses I haven't thought of?

Implementation/User Interface: Allow specifying both --include and --exclude patterns on the command line, which fill a common list (order does matter here). For each dir/file, check all patterns in the list. The action (exclude or include) of the last mattching pattern wins. The default action (which will also be used if neither include nor exclude patterns are specified) is "include".

This already contains a corner case: Should restic walk /home/user/work at all?

I think it should not. Instead, if this is a desirable behaviour, require the user to add a more specific pattern to signal that the excluded directory should also be walked, e.g. restic backup --exclude /home/user/work --include /home/user/work/**/*c. /home/user.

A better user interface would be to allow specifying a file to read the patterns from (--pattern-file or something like that). In this file, all lines starting with # are comments, empty lines ignored, all other lines must start with either + (include) or - (exclude) following a space character and a pattern. For the use case above a filter file would look like this:

# filter out everything from work, but include c source code files
- /home/user/work
+ /home/user/work/**/*.c

This issue can be closed once a definitive solution for having both include and exclude patterns is implemented.

@fw42
Copy link
Member

@fw42 fw42 commented Jul 18, 2015

Another possibility would be to read an exact list of files to backup from a file (or from stdin). Then people could just use whatever tools they want (find, grep, etc.) to build their list and "pipe" that list to restic.

@fw42
Copy link
Member

@fw42 fw42 commented Jul 18, 2015

I should read the whole text before posting a reply. Sorry.

@fd0
Copy link
Member Author

@fd0 fd0 commented Jul 18, 2015

This is a different issue, taking the list of files/dirs to backup from stdin instead of command line arguments. Do you think it's valuable to have that? If so, could you add an issue?

@fd0 fd0 mentioned this issue Jul 19, 2015
3 of 3 tasks complete
@fd0
Copy link
Member Author

@fd0 fd0 commented Nov 10, 2015

Just a reminder for myself: This issue is about include filters for backup, restore already has them.

@lathspell
Copy link

@lathspell lathspell commented Jan 24, 2016

You should maybe just copy rsync's --exclude/--include/--exclude-from syntax, they have 20 years of experience :-)
(at least please add some examples of the current syntax to the user guide as it's unclear if "/foo" and "foo" is thesame or if "*.c" is supported.

@fd0
Copy link
Member Author

@fd0 fd0 commented Jan 25, 2016

Ah, thanks for the comment, I've added an issue for the missing examples in the manual: #396

To be honest, I don't like the filter syntax from rsync at all, because the rules are way too complex. But we'll see what we can come up with.

@opennota
Copy link
Contributor

@opennota opennota commented Feb 3, 2017

Any updates? It is definitely a must-have option. Currently it is not even possible to read the file list to back up from the stdin:

$ find -name '*.go' | restic backup --files-from -
open -: no such file or directory

whereas it could be written as

restic backup --exclude '*' --include '*.go'
@fd0
Copy link
Member Author

@fd0 fd0 commented Feb 3, 2017

Uhm, reading the list of files from stdin can be achieved by calling restic as follows:

$ find -name '*.go' | restic backup --files-from /dev/stdin

If you like, I'd accept a PR that adds handling - for --files-from. :)

@fd0
Copy link
Member Author

@fd0 fd0 commented Feb 3, 2017

@opennota would you mind describing your use case? It'd be interesting for us.

@fd0
Copy link
Member Author

@fd0 fd0 commented Feb 3, 2017

The dash (-) issue is tracked as #769.

@opennota
Copy link
Contributor

@opennota opennota commented Feb 3, 2017

@fd0
Not much of a use case. I just want to back up only the files with certain extensions without using a temporary file for the list.

If you like, I'd accept a PR that adds handling - for --files-from. :)

I'll see what I can do.

@runfalk
Copy link

@runfalk runfalk commented Mar 19, 2018

You can kind of emulate this behavior using sed and named pipes:

restic --exclude-file <(sed -n 's/^- \(.*\)/\1/p' files.list) --files-from <(sed -n 's/^+ \(.*\)/\1/p' files.list)

Lines starting with - are excluded and those with + are included.

@dgasaway
Copy link

@dgasaway dgasaway commented Jul 17, 2018

I think a good model is what borg recently implemented for --pattern and --patterns-from
https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
Not so much the different style selectors, but the options to specify root paths, include rules, exclude rules, and no-recurse exclude rules in a file.

@blacksails
Copy link

@blacksails blacksails commented Jan 29, 2019

Why not just use the standard ignore format that .gitignore uses? eg.

# ignore everything
*
# include $HOME/.local
!$HOME/.local
@tnatanael
Copy link

@tnatanael tnatanael commented Apr 26, 2019

The --include is scheduled to be implemented?

Sound like --include and --exclude cannot be implemented together, or at least would be a hierarchy of precedence with one on other...

@vincentbernat
Copy link
Contributor

@vincentbernat vincentbernat commented Jun 14, 2019

Providing a list of files with --files-from doesn't totally solve the problem as the snapshots subcommand will display gigantic list of files and forget subcommand doesn't work as expected.

My use case is backing up my home and I have a list of paths with some exclusion and some exceptions for the exclusion. The base list of paths contains already a hundred of items. As everything is under $HOME, I would expect to be able to say something like --exclude=** --include=~/path1 --include=~/path2 --exclude=~/path2/something --exclude=*~. So, to determine if a path should be included it should be matched against each --exclude and --include in the right order and the last matching wins.

vincentbernat added a commit to vincentbernat/restic that referenced this issue Jun 14, 2019
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix restic#233
@vincentbernat vincentbernat linked a pull request that will close this issue Jun 14, 2019
7 of 7 tasks complete
vincentbernat added a commit to vincentbernat/restic that referenced this issue Jul 2, 2019
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix restic#233
@danielsharvey
Copy link

@danielsharvey danielsharvey commented Jul 3, 2019

I think @vincentbernat's PR is an effective solution to this requirement.

Summary: Allow gitignore-style negative patterns to specify exclude rules for both backup and restore.

I've been using it effectively for several weeks now.

Notably, this enables me to simplify my snapshots listing, which was quite verbose previously:

b951f6a2  2019-06-15 11:30:18  elvandar    manual      /Users/daniel/Desktop
                                                       /Users/daniel/Documents
                                                       <lots more...>

to:

d0c0bed1  2019-06-18 08:20:57  elvandar                /Users/daniel

Also, I've implemented an effective (not perfect but working) solution providing continuous backup using this feature (will share details shortly).

vincentbernat added a commit to vincentbernat/restic that referenced this issue Jul 3, 2019
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix restic#233
vincentbernat added a commit to vincentbernat/restic that referenced this issue Jun 27, 2020
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix restic#233
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

10 participants
You can’t perform that action at this time.