Adding a list of selectable exclude patterns

[torwag]

Hey,

I just struggle to identify possible exclude patterns and wondering if this would be a nice feature for vorta to assist the user.

Prominent candidates are

package managers
mail programs
browsers
compile workflows
in general, cache folders of certain programs

To make it easy for the users, instead of presenting an entire possible very large list, vorta could suggest, after a first backup, that certain folders might be excluded from the next backups as they are known as being irrelevant. For that, vorta would compare the backup folder tree with a list of given folders which are known to create large changes on each backup, but does not contain any valuable information.

Basically, you run the already existing exclusion pattern algorithm for all entries in that list and see if it creates a fit somewhere and if this folder contains data above a certain diskspace threshold, vorta propose it as potentially irrelevant.

If this folder list is maintained in an own file, it can be easily extended by individual PR.

[real-yfprojects]

I think #907 is the better way to solve the described problem. However, you name some good categories of exclude patterns that #907 could adopt.

[m3nu]

Exclude patterns for different OS and software is almost a project on its own. If someone can start a repo in a nice format, many projects could bene fit from it. Not sure this should be managed entirely in Vorta alone.

[torwag]

I agree with @m3nu that a repro with a nice parsable list of temporary files and folders for each program under each OS would be a real helper for many programs, not only vorta.

As for #907, it describes basically the same idea but in a reverse (traditional way). Presenting a list of possible exclusion patterns.
My suggestion to make this much more user-friendly would be that one carries out one backup (or mimic to do so, and test all available patterns against the given source folder and file structure. Defined by a threshold and the size of the files/folders, it compiles a list from large to small disk space savings and ask the user to make selections which folders are not really necessary in a backup.

If this mechanism could be rerun also between backups as diff, a user could quickly identify that the main part of an hourly backup is due to some temp files and select to exclude them in future backups.
Such a diff list in between backups could be as short as something like (only folders or files are shown which hit a predefined exlude pattern and hence are known to be "safe" to be excluded):

/tmp 8G
/home/foo/.cache/ 4G
/.snapshot-programmXXX 2G
/home/foo/project/compile/cache 1G

The user would directly spot those candidates instead of wondering if //$HOME/.?[Cc]ache/ would actually have any hits.

[real-yfprojects]

The user would directly spot those candidates instead of wondering if //$HOME/.?[Cc]ache/ would actually have any hits.

Is this your actual problem or do you need vorta to assist you in finding large folders/files that can be excluded in a backup?

The diff feature already allows to see the sizes of changes and will soon be able to sort by these sizes. There are also directory analysation tools that can be used to visualize the disc space of contents in your file system (e.g. filelight).

[torwag]

Well, the identification on one side, but also the "translation" into a fitting excluding pattern on the other side. This isn't a big problem for me, I just believe it would be beneficial for many users.

How about the possibility to select folders in the diff-view and add them to the exclusion pattern. Users could run backups and if they tend to be surprisingly large, analyse with the diff tool, what is the cause of it and if new temp or cache folders are identified by the user, he can simply mark them as "exclude this folder/file for new backups". This will create an entry in the "exclude pattern list" and the next backups should be much smaller.

This proposal is basically the above idea, replacing the AI/algorithm part to identify unneeded large file changes by wetware (aka brain).