Allow mixing include and exclude patterns for backup and restore #233

Open
fd0 opened this Issue Jul 18, 2015 · 13 comments

Comments

Projects
None yet
6 participants
@fd0
Member

fd0 commented Jul 18, 2015

This is a followup issue for #226. Up to now it is only possible to specify exclude patterns for backup and restore.

There are uses cases that aren't covered, for example: A user would like to backup her home directory at /home/user, excluding anything from the directory work except C source files.

Are there any other uses I haven't thought of?

Implementation/User Interface: Allow specifying both --include and --exclude patterns on the command line, which fill a common list (order does matter here). For each dir/file, check all patterns in the list. The action (exclude or include) of the last mattching pattern wins. The default action (which will also be used if neither include nor exclude patterns are specified) is "include".

This already contains a corner case: Should restic walk /home/user/work at all?

I think it should not. Instead, if this is a desirable behaviour, require the user to add a more specific pattern to signal that the excluded directory should also be walked, e.g. restic backup --exclude /home/user/work --include /home/user/work/**/*c. /home/user.

A better user interface would be to allow specifying a file to read the patterns from (--pattern-file or something like that). In this file, all lines starting with # are comments, empty lines ignored, all other lines must start with either + (include) or - (exclude) following a space character and a pattern. For the use case above a filter file would look like this:

# filter out everything from work, but include c source code files
- /home/user/work
+ /home/user/work/**/*.c

This issue can be closed once a definitive solution for having both include and exclude patterns is implemented.

Also add back the Test removed in a4530c928091c58f399f74d8515f28012726b717 TestRestoreNoMetadataOnIgnoredIntermediateDirs.

@fw42

This comment has been minimized.

Show comment
Hide comment
@fw42

fw42 Jul 18, 2015

Member

Another possibility would be to read an exact list of files to backup from a file (or from stdin). Then people could just use whatever tools they want (find, grep, etc.) to build their list and "pipe" that list to restic.

Member

fw42 commented Jul 18, 2015

Another possibility would be to read an exact list of files to backup from a file (or from stdin). Then people could just use whatever tools they want (find, grep, etc.) to build their list and "pipe" that list to restic.

@fw42

This comment has been minimized.

Show comment
Hide comment
@fw42

fw42 Jul 18, 2015

Member

I should read the whole text before posting a reply. Sorry.

Member

fw42 commented Jul 18, 2015

I should read the whole text before posting a reply. Sorry.

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Jul 18, 2015

Member

This is a different issue, taking the list of files/dirs to backup from stdin instead of command line arguments. Do you think it's valuable to have that? If so, could you add an issue?

Member

fd0 commented Jul 18, 2015

This is a different issue, taking the list of files/dirs to backup from stdin instead of command line arguments. Do you think it's valuable to have that? If so, could you add an issue?

@fd0 fd0 referenced this issue Jul 19, 2015

Merged

Add filter implementation for files #231

3 of 3 tasks complete
@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Nov 10, 2015

Member

Just a reminder for myself: This issue is about include filters for backup, restore already has them.

Member

fd0 commented Nov 10, 2015

Just a reminder for myself: This issue is about include filters for backup, restore already has them.

@lathspell

This comment has been minimized.

Show comment
Hide comment
@lathspell

lathspell Jan 24, 2016

You should maybe just copy rsync's --exclude/--include/--exclude-from syntax, they have 20 years of experience :-)
(at least please add some examples of the current syntax to the user guide as it's unclear if "/foo" and "foo" is thesame or if "*.c" is supported.

You should maybe just copy rsync's --exclude/--include/--exclude-from syntax, they have 20 years of experience :-)
(at least please add some examples of the current syntax to the user guide as it's unclear if "/foo" and "foo" is thesame or if "*.c" is supported.

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Jan 25, 2016

Member

Ah, thanks for the comment, I've added an issue for the missing examples in the manual: #396

To be honest, I don't like the filter syntax from rsync at all, because the rules are way too complex. But we'll see what we can come up with.

Member

fd0 commented Jan 25, 2016

Ah, thanks for the comment, I've added an issue for the missing examples in the manual: #396

To be honest, I don't like the filter syntax from rsync at all, because the rules are way too complex. But we'll see what we can come up with.

@opennota

This comment has been minimized.

Show comment
Hide comment
@opennota

opennota Feb 3, 2017

Contributor

Any updates? It is definitely a must-have option. Currently it is not even possible to read the file list to back up from the stdin:

$ find -name '*.go' | restic backup --files-from -
open -: no such file or directory

whereas it could be written as

restic backup --exclude '*' --include '*.go'
Contributor

opennota commented Feb 3, 2017

Any updates? It is definitely a must-have option. Currently it is not even possible to read the file list to back up from the stdin:

$ find -name '*.go' | restic backup --files-from -
open -: no such file or directory

whereas it could be written as

restic backup --exclude '*' --include '*.go'
@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Feb 3, 2017

Member

Uhm, reading the list of files from stdin can be achieved by calling restic as follows:

$ find -name '*.go' | restic backup --files-from /dev/stdin

If you like, I'd accept a PR that adds handling - for --files-from. :)

Member

fd0 commented Feb 3, 2017

Uhm, reading the list of files from stdin can be achieved by calling restic as follows:

$ find -name '*.go' | restic backup --files-from /dev/stdin

If you like, I'd accept a PR that adds handling - for --files-from. :)

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Feb 3, 2017

Member

@opennota would you mind describing your use case? It'd be interesting for us.

Member

fd0 commented Feb 3, 2017

@opennota would you mind describing your use case? It'd be interesting for us.

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Feb 3, 2017

Member

The dash (-) issue is tracked as #769.

Member

fd0 commented Feb 3, 2017

The dash (-) issue is tracked as #769.

@opennota

This comment has been minimized.

Show comment
Hide comment
@opennota

opennota Feb 3, 2017

Contributor

@fd0
Not much of a use case. I just want to back up only the files with certain extensions without using a temporary file for the list.

If you like, I'd accept a PR that adds handling - for --files-from. :)

I'll see what I can do.

Contributor

opennota commented Feb 3, 2017

@fd0
Not much of a use case. I just want to back up only the files with certain extensions without using a temporary file for the list.

If you like, I'd accept a PR that adds handling - for --files-from. :)

I'll see what I can do.

@runfalk

This comment has been minimized.

Show comment
Hide comment
@runfalk

runfalk Mar 19, 2018

You can kind of emulate this behavior using sed and named pipes:

restic --exclude-file <(sed -n 's/^- \(.*\)/\1/p' files.list) --files-from <(sed -n 's/^+ \(.*\)/\1/p' files.list)

Lines starting with - are excluded and those with + are included.

runfalk commented Mar 19, 2018

You can kind of emulate this behavior using sed and named pipes:

restic --exclude-file <(sed -n 's/^- \(.*\)/\1/p' files.list) --files-from <(sed -n 's/^+ \(.*\)/\1/p' files.list)

Lines starting with - are excluded and those with + are included.

@dgasaway

This comment has been minimized.

Show comment
Hide comment
@dgasaway

dgasaway Jul 17, 2018

I think a good model is what borg recently implemented for --pattern and --patterns-from
https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
Not so much the different style selectors, but the options to specify root paths, include rules, exclude rules, and no-recurse exclude rules in a file.

I think a good model is what borg recently implemented for --pattern and --patterns-from
https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
Not so much the different style selectors, but the options to specify root paths, include rules, exclude rules, and no-recurse exclude rules in a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment