Exclude patterns GUI improvements

[m3nu]

Just noticed that Nextcloud (desktop client) comes with a nice list of default exclusions. Maybe something for Vorta to assimilate.

I agree that one simple kind of patterns would be enough.

Originally posted by @m3nu in #838 (comment)

[erAck]

BackInTime also has a sensible set of default excludes:

• .gvfs
• .cache/*
• .thumbnails*
• [Tt]rash*
• .backup
• *~
• .dropbox*
• /proc/*
• /sys/*
• /dev/*
• /run/*
• /etc/mtab
• /var/cache/apt/archives/*.deb
• lost+found/*
• /tmp/*
• /var/tmp/*
• /var/backups/*
• .Private

[m3nu]

We discussed adding defaults before and the feedback wasn't good due to the large variety of OS and use cases.

So if we add defaults, they will be OS-specific and disabled by default.

[m3nu]

Suggested by @Wikinaut in #970

I cannot find that borg option --exclude-caches to be used by vorta. Was that intended?

See https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=exclude%20caches

I suggest to add this as a vorta option which is active by default.

[m3nu]

Getting close to starting this feature. Was thinking about this a bit more and would want those features:

• Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".
• Currently I use 2 groups of exclusions: specific paths (e.g. ~/Library/BlahBlah/Cache and patterns that should be matched everywhere (e.g. */.git)
• Maybe we also want "Exclude folders with this file"
• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

Anything to add? @Hofer-Julian @bastiencyr ?

[Hofer-Julian]

Sounds good.
One idea:

• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

How about allowing comments with #?
In that case you don't need an extra field for descriptions and the export format would be a list of patterns and comments instead of a csv.

[m3nu]

I like the idea of making the import/export file a valid patterns file.

In the UI, I'd prefer to have pattern and commend separate. Else we literally have a free text box again.

[bastiencyr]

Souds good too.

Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".

For naming, maybe some more meaningful names for users like "User Temporary Files", "Dev Irrelevant Files". Its more user orientated and may not know what a cache is.

[m3nu]

macOS via arkTanlis on Reddit.

(Some wildcards used here are not appropriate for the default fnmatch patterns we use.)

~/.Trash
**.DS_Store
~/Library/Caches
~/Library/Logs
**.cache
~/.npm
~/Library/Containers
**.git
**node_modules

[m3nu]

Adding my own excludes:

*/.DS_Store
*/._sync_*
*.pyc
*.mp4
*.mkv
*/.~*
*-journal
*/.owncloudsync.log
*/.cache
*/__pycache__
~/Pictures/20*
~/Pictures/Phone

[m3nu]

Some more inspiration from Back in Time. They probably keep a list for Linux.

[real-yfprojects]

I sketched a first GUI mock-up. What do you think?

[m3nu]

I think it's great. 🙌 Not much to add.

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

[real-yfprojects]

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

There patterns that can be grouped together with a name. They can be toggled through the GUI or by commenting them out in raw text.
The predefined pattern groups that have tags (if they are app, os specific,...) and possibly a description. There is a search bar for searching through the patterns/groups by group name and tags as well as pattern contents.

We can integrate online sources for patterns like gitignore.io or github/gitignore for excluding different development files.

[real-yfprojects]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

[m3nu]

That's not very flexible. E.g. to exclude a type of files.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

That specific feature is not released yet.
But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

[m3nu]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

[real-yfprojects]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

Yes, that would be a good start. However the user should be able to toggle single patterns, edit the predefined patterns and add own patterns. This could be realised in two database tables:
PatternGroup and Pattern. However my idea was getting the data from a raw text file/entry the user can edit.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

Thanks. I like that the mockup shows the size of the excluded files.

[fooness]

Hello everyone. Could you please point me to the current list of different exclude defaults/suggestions? Maybe I missed the link here, or did search for the wrong keywords. I thought I could look the exclusions up here in the repo, without installing Vorta (as I’m basically using borg without GUI for my macOS backups).

---

Here’s what I came up with for now; it’s a mixture of Arq’s exclusion list as well as the suggestions from the previously linked reddit thread and from the posts in this very GitHub issue … comments and feedback very much appreciated; this is macOS-specific.

**/.*.swp
**/.DS_Store
**/.VolumeIcon.icns

# not quite sure what the following two exclusions are for
**/.file
**/.vol

**/.DocumentRevisions-V100/
**/.MobileBackups/
**/.MobileBackups.trash/
**/.Spotlight-V100/
**/.TemporaryItems/
**/.Trash/
**/.Trashes/

**/.cache/
**/.fseventsd/
**/.npm/

**/Library/**/Cache/
**/Library/**/Caches/
**/Library/**/Containers/
**/Library/**/CrashReporter/
**/Library/**/DerivedData/
**/Library/**/Developer/
**/Library/**/Metadata/
**/Library/**/MobileSync/
**/Library/**/Logs/

**/iTunes/

**/build/
**/dist/
**/node_modules/
**/tmp/
**/vendor/

/System/
/Users/Shared/
/Volumes/
/bin/
/cores/
/dev/
/etc/
/home/
/macOS Install Data/
#/nix/
#/opt/
/private/
/sbin/
/tmp/
#/usr/
/var/

# comment: /nix/ + /opt/ + /usr/ maybe should not be excluded due to e.g. homebrew …

[m3nu]

Thanks for sharing! We don't ship a default list yet. Still work in progress. Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

[fooness]

Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

Yes, the definitely are; been looking for the macOS-specific list.

We don't ship a default list yet. Still work in progress.

That’s the part I obviously have missed reading through this issue; will keep an eye on future updates.

PS: I’ll be doing some more research on this matter, and in case of any new insights, I’ll be editing/updating my list above.

[real-yfprojects]

Is it something like this?

Yes, exactly.

I feel like the presets should just be there to assist the user in getting pre-defined patterns. If they really want to see why a pattern is there they can just look in the Raw tab. I wanted to keep the Patterns tab less cluttered and the simplest thing a user would want.

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

[diivi]

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too.
But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

[real-yfprojects]

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too.

Ah, that wasn't clear since your mockup didn't contain patterns only.

But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

I think there will be a lot of patterns pretty fast. There should be a way to organize them.

[diivi]

Ah, that wasn't clear since your mockup didn't contain patterns only.

Can you check now? I went back to the approach of keeping patterns and presets separately, with a line separating them. To view all patterns associated with a preset, you can view the raw tab.
This will only work when the preset is added as is, without modification, otherwise the preset becomes None and all patterns are added separately.

[real-yfprojects]

I went back to the approach of keeping patterns and presets separately, with a line separating them. To view all patterns associated with a preset, you can view the raw tab.

There should be a way to highlight the patterns in the raw view.

[diivi]

Highlight which patterns?
And what kind of highlight?
Highlight after search (planning to do that), or highlight = selecting text (can do that).

[real-yfprojects]

Highlight which patterns?

The patterns associated with a preset. Else you have to search them manually.

[diivi]

I made some more progress, thanks to suggestions from @real-yfprojects and @erAck , this is what the UI looks like currently:



Again, the designs are available here for anyone to comment/give feedback - https://www.figma.com/file/Y70aJFOpJ1VT59lWzUZEoQ/Exclude-GUI?node-id=0-1&t=yONNiNodanw9dnE4-0.

[m3nu]

You should consider the data structure before doing the GUI.

First there could be groups that enable multiple rules. Like macOS cache files. Precise content would be hidden and may change during updates. So having presets as simple text and then putting everything in a large text field isn't the best way.

Then the user may add some raw rules. Those would be merged with the rules we ship as presets.

Presets could be just checkbox + name + details as tooltip or collapsible.

Custom rules could be what you have as patterns now. Text + checkbox.

Raw is useful to quickly add many existing rules.

Finally, Preview would merge it all together to help debugging.

[diivi]

• Okay, I'll change the current Patterns tab to Custom and make that accept only custom exclusion rules.
• I'll add a presets tab that has all the grouped rules, and will be similar to the Patterns tab, but will have preset names instead of rules and a tooltip to show the details. Or, I'll make it collapsible to view all excluded files (I can do this on Figma, but will need help in making a collapsible list on qt).
• Raw remains the same
• The add dialog will be a single input text field to add a single custom exclusion rule?

Finally, Preview would merge it all together to help debugging.

Do you mean there should be another tab? What will this tab show? I think the check input below is sufficient for debugging.
Also, the raw view will already show all the rules inside a preset group.

[m3nu]

Preview tab could show all rules combined how they are passed to Borg later.

Raw would be just for user input. If you combine both, how do you know what the user edited and what's from our own presets?

[diivi]

Okay, and how do you recommend the separation of user added rules and vorta preset rules be done in the preview tab?

[diivi]

Also, I'm concerned that having four tabs and too many features may be counterproductive, as it could make the interface more difficult for novice users to navigate.
I agree with your first point about the presets getting updated and a simple text field might not be the best solution. But I'm not so sure about adding so many things :/. Maybe we can just change the way presets are handled and added?

[real-yfprojects]

I like @m3nu suggestion. What about having two tabs: Patterns and Raw. The pattern tab shows the patterns and collapsible pattern groups including added presets. The raw tab shows the custom patterns at the beginning and the preset patterns appended to the end. However the section of the textedit with preset patterns is greyed out and can't be edited. There won't be any merging of preset and non preset patterns, borg can handle duplicates.

[diivi]

Cool, if that's what we decide, then I'll edit my design to accomodate groups and single rules both in the patterns tab.
What about the add tab? Do you think I should change something? Maybe disable the textfield when a preset is selected? Or, with the current approach:

• if it is edited then change the preset to none and treat them as seperate rules
• if an unedited preset is added, we can keep track of it and update it easily.

[vv111y]

This is great that you are doing this. While researching I found this repo for list of excludes for linux /home with rsync, it should be good for this feature
https://github.com/rubo77/rsync-homedir-excludes

[diivi]

Thankyou! This is really helpful :)

[diivi]

Thanks to everyone who participated in this thread!

The current iteration after community review and feedback from the maintainers can be found here - https://www.figma.com/file/Y70aJFOpJ1VT59lWzUZEoQ/Exclude-GUI?type=design&node-id=0-1&t=vXY1HiJ152wU6FhF-0.

Attaching screenshots here too:

This is one of my GSoC projects, and I'll start coding in a few weeks. Please let me know if anyone has other suggestions/improvements that I can look into before that!

[m3nu]

Looks good and what I imagined really. Minor thoughts:

• For the Preview tab it should be clear that the text isn't editable. Else I imagine it's the same as the Raw tab.
• The text box at the bottom with "Placeholder" is for testing? Writing a full path may not be feasible and we need a file selector there? Or leave out the feature for now.
• Custom tab: We want to add a comment/notes for custom patterns?
• Search feature: not sure that's useful. Maybe just in the Custom tab for filtering out lines? Not sure how easy it is to select text in Qt in a textbox.

[sammcj]

It might be good to include a field that you can provide exclusion lists to, it's not currently clear if the UI supports .lst files or not.

e.g, the vorta equivalent of running borg create --patterns-from=borg-exclude-macos.lst

Where the .lst contains items such as:

# Siri
! **/Library/Assistant/SiriAnalytics.db
! **/Library/Assistant/SiriSyncItems.db
! **/Library/Assistant/SiriVocabulary/sv.db

# Logs and Monitoring
- **/Logs/

[real-yfprojects]

The current mockup doesn't allow patterns afaik but only excludes. There is a TextEdit for pasting in your own excludes e.g. from a different file.

[diivi]

Hey, I have started work on this feature and will soon get started with the Presets tab. These presets will be shipped by Vorta and you'll just have to select some presets and all patterns included in that preset will be excluded from the created archive!

I need some feedback/suggestions for these presets though, the format I need is -

Preset Name
Description
/pattern1
/pattern2
...

Preset 2 Name
Description
/pattern
...

So if anyone has a grouped list (or even one set of patterns), like this - #907 (comment) it'd be helpful to get things started on this feature. Thanks!

P.S. you can add your preset by opening an issue after this feature is shipped too. These presets will be a json file in the Vorta codebase itself.

[m3nu]

Parsing this will be a minefield and we may want more metadata. Would use a dict/json format.

name, desc, os, rules format, list of actual rules, maybe?

[diivi]

Oh yeah, the implementation itself will be a json, I just meant to list what's needed 😅

[sammcj]

If it helps these are the excludes I use on macOS:

**/.minikube/
**/.npm/
**/.platformio
**/.rbenv
**/.rvm
**/.timemachine/
**/.Trash
**/.wine
**/*.DS_Store
**/*.git
**/*.temp
**/*.tmp
**/*diffusionbee/
**/*docker-desktop/
**/*lima/
**/*nstall*.dmg
**/*universal*.dmg
**/*x64*.dmg
**/docker-volumes/
**/iTunes/iTunes Media/podcasts/
**/iTunes/iTunes Music/
**/miniconda3/
**/node_modules/
**/nomic.ai/
**/temp/
**/tmp/
**/Virtual Machines/
**/Xcode.app/
/Applications
/Users/**/_NoSync_NoBackup/
/Users/*/.asdf
/Users/*/.cache
/Users/*/.cargo
/Users/*/.colima
/Users/*/.docker
/Users/*/.platformio
/Users/*/Applications
/Users/*/container_mount/
/Users/*/Documents/Adobe/
/Users/*/git/
/Users/*/go/
/Users/*/Library/Application Support/Autodesk/
/Users/*/Library/Application Support/Caches/
/Users/*/Library/Application Support/Code/Cache*/
/Users/*/Library/Application Support/coreMLCache/
/Users/*/Library/Application Support/Firefox/Profiles/*/storage/temporary/
/Users/*/Library/Application Support/Firefox/Profiles/*/storage/to-be-removed/
/Users/*/Library/Application Support/Firefox/Profiles/storage/default/
/Users/*/Library/Application Support/fnm/
/Users/*/Library/Application Support/Google/Chrome/*Cache*/
/Users/*/Library/Application Support/Google/Chrome/File System/
/Users/*/Library/Application Support/Google/Chrome/IndexDB/
/Users/*/Library/Application Support/Google/Chrome/Safe Browsing/
/Users/*/Library/Application Support/Google/Chrome/Service Worker/
/Users/*/Library/Arduino15/
/Users/*/Library/Caches/
/Users/*/Library/Containers/com.apple.AMPArtworkAgent/
/Users/*/Library/Containers/com.bitdefender.*/
/Users/*/Library/Containers/com.microsoft.*/
/Users/*/Library/Containers/F807763E-5B70-49A2-B0B0-E13291698A23/
/Users/*/Library/Developer/
/Users/*/Library/Metadata/
/Users/*/Library/Saved Application State/
/Users/*/Music/
/Volumes/*timemachine*/
/Volumes/*TimeMachine*/

[jakesmolka]

Sounds awesome @diivi 👍 Looking forward using this feature.
Please also check https://github.com/SterlingHooten/borg-backup-exclusions-macos not just for the lists, but also for the knowledge – if that helps.

[m3nu]

Love all the input, @jakesmolka and @sammcj . Will be great to bundle this knowledge with Vorta. We just need to group it all a bit.

@diivi , you see your feature is already very popular. I suggest to add the most obvious input from here to the first version, but also prioritize getting something usable out of the door and then add more exclusion rules over time.