Revise default backup filter UI

[Pectojin]

Now that backup filters actually work when defined in the Web GUI we should revise the default filters to make sure other users aren't hit by similar filter issues as I was:
https://forum.duplicati.com/t/release-2-0-2-19-canary-2018-02-12/2455/6

Additionally I think we should improve the UI to display more information on what the default filters actually do.

I think it's a show stopper for an experimental release to risk many people upgrading and not discovering that parts of their backups are now filtered aggressively.

[renestach]

I completely agree that we should explain more what the filters actually do. Do you think a short description is sufficient like "These filters exclude system files from your backup. Please check the content of your backup when one of the filters is activated."

[Pectojin]

I'm not sure. It seems to be a little more complicated than just being system files https://forum.duplicati.com/t/release-2-0-2-19-canary-2018-02-12/2455/14
That description wouldn't let me know that all of /var including /var/www isn't backed up.

There are multiple rules that I think are questionable in this list and even after fixing those it may still be an issue about what people think of each rule.

The UI doesn't have to display them in-line, but I think it's unfair to require a user to make a full backup and check the contents of all folders before they can "know" what the filter does.

[renestach]

I cannot remember when we introduced this and what the use case behind those OS-specific filters was. Were they introduced so that a simple backup of / or C:\ would turn into a backup that only contained user data? Or is their purpose to remove sneaky files from the backup like the famous 'desktop.ini' on Windows?

Before we revise the filters, we have to define what the exact use case is. If we have that definition already, please tell me :-)

[tygill]

I was the one to initially add it, and my initial goal was mostly to remove things that aren't really important to be backed up (caches, temporary files, etc.), and to prevent errors attempting to back up system files that don't need to be backed up which might not be normally accessible (e.g., System Volume Information, $RECYCLE.BIN). One specific example is things like the Google Chrome cache in the AppData folder - I want to backup my entire user folder, but don't need to keep a running update of what happened to be in my Internet cache that day.

Originally, I was also thinking that system files (OS, program files, etc.) should be skipped, but especially after the recent discussions I'm much more inclined towards not filtering those (and letting the user choose to back them up by selecting those folders as sources instead).

I just pushed a commit to my branch that removes a number of the default filters for folders that might have things that should be backed up. @Pectojin - Does it remove most of the rules that you think are still questionable?

(It's more closely related to what Crashplan does, as most of the filters are now the 'Admin filters' from here, while most of the things that Crashplan just hides by default are no longer filtered.)

[Pectojin]

Heh, @tygill beat me to commenting by like 40 seconds and made my comment irrelevant :)

The new commit is looking good. It addresses all my original concerns for my own backup.

You removed some filters that I might not have removed - I'd never back up /lib or /bin - but on the other hand maybe someone else would.

Overall I'd say this would bring this issue close to resolved for my own purposes, but I don't know if anyone else has thoughts on Windows/Mac filters, as I didn't use those.

I'd still like the option for the user to view a list of filters applied by the defaults, though. But maybe it's less important if no one is affected negatively by the new filter

[renestach]

Even if the rules fix your issues/concerns, users will not understand what they will get when they select one of these options. And from checking the rules, I have no clue how to explain in simple words what is going to happen.

I also get the impression, that none of the OS-specific filters will ever be perfect for all users. Why not use the filter rules as suggestions that the users can pick from? That way we can offer rules for specific purposes.

The UI could then look like this:

As a nice side effect: We give users examples how filters are defined.

[Pectojin]

Ah, that might be a good way to show it. Actual tangible results that can be inspected and tweaked.

It shouldn't be terribly difficult to implement but I'm thinking there might be some issues defining when the box is "checked". If I check it and remove 9 out of 10 rules, is it still "checked"?

And do we remove the rules when unchecking? As long as they haven't been changed?

[Pectojin]

Maybe we should make a new issue for this frontend and ship it in next canary/experimental?

It won't affect users who already set up their backup since they won't look at it, and it wasn't displayed before so nobody is "losing" functionality.

[tygill]

@Pectojin - I see /lib and /bin as very similar to Program Files on Windows - while it's likely that most programs can probably be reinstalled fairly easily, it's possible some users might have things installed there by hand or in a way that they want to back them up.

@renestach - One difference with doing it that way (offering suggestions that the user opts-in to) is that if the filters change in the future (either to add or remove or clarify a rule), the user wouldn't get that change without editing the backup and looking at the new suggestions. I can see how that could be considered both a good and a bad thing, but it is a difference. I would also want to make sure that default filters are relatively straightforward to use from the commandline.

[renestach]

I am not sure if you got my idea. There are no check boxes anymore. Actually we define a list with sections (Windows, Linux, Mac). Within those sections we have specific filter rules. The users can use the dropdown boxes to first pick a section and second one of the rules within this section. The (+) button will then add the selected rule to the list of filters at the top. This filter can be modified or deleted then.

Actually, it is just a list of examples and recommendations to choose and learn from.

[renestach]

@tygill I don't think we should change rulesets for existing updates at any time unless we have a very, very precise definition of what the ruleset exactly does (we do not have this - the list looks quity arbitrary). We are discussing this feature now because someone complained that the behaviour has changed when the feature was fixed.

For our personal peace of mind I recommend to change the feature in a way that we suggest filters that make sense and users have to pick them manually. No update will change any backup. Users get exactly what they think they need. And they learn how filters work.

Last but not least: It would deactivate all the filters again, so that we do not have to be careful about the next update.

[renestach]

Another suggestion:

[kees-z]

@renestach your suggestions could work, but have some backdraws.
Suppose I'm on Windows and want to exclude Windows related trash. In your screenshot I see a pull-down menu with 2 options, something like /etc/* and /var/*. But the Windows exclusion list looks something like this:

• You suggest adding OS specific exclusions one by one. The average Windows user would like to apply almost all exclusions. There are hundreds of mouseclicks required, the same action has to be repeated many times.
• Just listing the exclusions will not end up in an easy to read list (see screenshot above). Finding the exclusions you need is difficult.
• Not all expressions are /var/* and /etc/*. What would the average user think of an entry like (?i).*/Google/Chrome/User Data/Default/Cookies(-journal)?", "(?i).*/Safari/Library/Caches/.*?

I like the checkbox approach more. Choose an OS and open the pull-down menu, just like your first suggestion. However, the pull-down menu doesn't list the expressions itself, but human-readable descriptions (System Volume Information, Windows Registry, Cache files, Recycle bin, etc).
Hovering over an option (or alternatively a small (i) symbol at the end of each line) reveals the expression(s) of that line.
Each entry in the pull-down menu has a checkbox, selected by default. You can disable if you don't want to exclude this expression.

Generally, I agree being reserved with OS specific exclusions. They should be limited to

• Files and folders that generate errors and/or result in a failed backup (Windows registry, System volume information)
• Files and folders that are 100% (99.9999%) known to be unneeded (cache files, desktop.ini).

Other files like application settings shouldn't be excluded by default, they do not harm and the user can exclude them himself. Maybe they can be listed in the pull-down menu, but disabled by default.

There's one problem with the "checkbox approach": how to parse this from the GUI to the command line, I'm not sure how to deal with that.
One option could to add all expressions to the commandline internally, but hide all built-in OS specific filters from the custom filter list view. So there will be the current list of user-defined custom filters, an OS selection pull-down to select a default exclusion list and next to it a pull-down containing a list with human-readable exclusions.

[tygill]

@kees-z just beat me to expressing some of the same points I was thinking.

@renestach - Essentially it sounds like you're advocating moving filters out of the internal code completely, and turning them into a UI feature, correct? This would make using default filters from the command line more difficult (though I suppose we could still keep the existing flag and behavior, but change the UI to not use it directly).

Continuing on kees-z's points, one thought is that we could break the filters down into more granular groups than just 'Windows', 'Mac', 'Linux', and each of these groups could have it's own checkbox (and possible a (i) button / tooltip showing the filters it contains, for better discoverability for power-users):

• All
  • Browser Caches (common caches for Chrome, Firefox, Safari, etc.)
• Windows
  • File System Metadata (/$RECYCLE.BIN/, /Recycler, /System Volume Information, etc.)
  • Caches (browser caches, Temporary Internet Files, Thumbs.db, etc.)
  • Temporary Files (/Windows/Temp*, /AppData/Temp*, *.tmp, etc.)
  • Program Files (/Program Files/, /Program Files (x86)/)
  • Windows Files (/Windows, /Microsoft*/Windows/*.log, etc.)

...

Assigning each filter to a group may not be terribly straightforward, but it would probably be a good exercise in making sure the filters are indeed useful. It would also make sure it is clear to the user what each is excluding. Ideally they wouldn't change in the future, but this way we would open up the possibility of adding new groups. This could also be done using the existing design and infrastructure, which would make the filter groups still easily used on the command line.

On a related note, I also remembered that @kenkendk added support (on Windows) to add custom default filters via the registry (based on this thread, it looks like there's a standard registry key for indicating files that shouldn't be backed up which it uses). If we go the route of making this a UI thing, then that feature would be disabled as well.

[kenkendk]

I see two different users: pro users that really do not like us changing the backup settings after they have verified that it works as they want. The other is the "casual" user, who is likely to just add "C:" without excluding anything "just in case".

Traditionally, we have tried to cater to both by adding commandline options, where the default is what we expect casual users to want, and I think we can do the same here.

I do not see any case where one would want to apply options for another OS. It makes no sense to exclude Windows paths on MacOS for instance.

I think this feature works better if we divide the filters into "groups", such as: log files, temporary files, system files, program files, etc.

The exact filter rules needs to be documented somewhere, but they are different between OS'es and can change with each release.

We can allow a group as a normal filter, like:

-*.txt
-{system}
-[.*\.dat]

This would allow interleaving the groups with normal filters and thus override rules, and is also easy to merge into the UI.

Would this resolve the concerns?

[kees-z]

Traditionally, we have tried to cater to both by adding commandline options, where the default is what we expect casual users to want, and I think we can do the same here.

I suppose any exclusion is disabled by default and a common set of files/folders is excluded after checking an exclusion group. Just like it is now with System, files, Temporary files and Hidden files.

I do not see any case where one would want to apply options for another OS. It makes no sense to exclude Windows paths on MacOS for instance.

Agreed. It also makes it a bit easier to present it in the GUI.

I think this feature works better if we divide the filters into "groups", such as: log files, temporary files, system files, program files, etc.

I guess that would work pretty well and gives more freedom to the user what to in/exclude without specifying tons of exclusions.

The exact filter rules needs to be documented somewhere, but they are different between OS'es and can change with each release.

I think they should be available from within the GUI, by hovering over the group name or clicking a small info-symbol. In the current version, when I check "apply default filters for Windows", I have no idea what's actually excluded. I would have to Google that, but if filter sets could change, I still wouldn't know what's excluded. This lack of information was the main reason for the OP's problems.

We can allow a group as a normal filter, like:
...
This would allow interleaving the groups with normal filters and thus override rules, and is also easy to merge into the UI.

Suppose there is an exclusion group "TempFiles" that applies the filters -*.tmp, -C:\Temp\* and -C:\Windows\Temp\ and an exclusion group "LogFiles" that excludes several files and folders with logfiles.

• These filter sets could be activated by just clicking a check box in the GUI, but what about the commandline? Something like --default-filters=TempFiles;LogFiles? What about the current values Windows, OSX, Linux and All?. All could be modified to include all groups and support for Windows, OSX and Linux could be dropped, but this will break backward compatibility.
• If I want to exclude temporary files, except C:\Temp because it contains files of a project I'm working on, how would this overriding of rules work? In the current implementation it's not possible to re-include already excluded files/folders.

[kenkendk]

I think they should be available from within the GUI,

Yes, preferably from the commandline as well.

These filter sets could be activated by just clicking a check box in the GUI, but what about the commandline? Something like --default-filters=TempFiles;LogFiles?
If I want to exclude temporary files, except C:\Temp

That is why I want to move away from the --default-filters, it breaks the filter flexibility by not allowing you to set the order. If we use the filter groups, you can override the filters by inserting an include before the filter.

What about the current values Windows, OSX, Linux and All?. All could be modified to include all groups and support for Windows, OSX and Linux could be dropped, but this will break backward compatibility.

We can allow the current options, but mark them deprecated. I think that is fine as the feature never went into an experimental release.

If we want to support something with more ease of use, we could add the groups to the list and only deprecated the OS names.
We then need to add such filters to the end of the filter list, which allows the explicit include mentioned above, but breaks if the filter list contains -*, as that will take precedence over the added groups.

[Pectojin]

I think my original request has been fulfilled, but then we got caught up in discussing UI changes.

I don't think those UI changes are necessary for an experimental release, but we've got a lot of good discussion here. I'll take the milestone tag off and update the title to be more UI-centric :)

[tygill]

Sounds good. In the meantime, I just opened a pull request for the change I mentioned above to remove some of the more problematic filters for now.

[renestach]

After discussing this feature, I think we should redesign how it works. Other than what I suggested previously, we might introduce specific filters (that contain a defined set of filter rules) in the dropdown. Here, we can also explain what the filters do. See my mockup to understand what I mean.

We still have to define what the filters are doing though. Caches are obvious. We could also exclude system-clutter on network drives and others.

[Pectojin]

I like that idea. Gives more control and more transparency but at no cost to ease of use.

[kenkendk]

I have now implemented the code for this feature in a branch:
https://github.com/duplicati/duplicati/tree/feature/filter_groups

The major changes are in:
https://github.com/duplicati/duplicati/blob/282c70f75f954c70e471dda3c21e7a54869bcbbd/Duplicati/Library/Utility/DefaultFilters.cs

For now I have removed the Windows/OSX/Linux options, but we can consider re-adding them in the parsing code, such that they are individually equivalent to "Default":

duplicati/Duplicati/Library/Utility/DefaultFilters.cs

Line 81 in 282c70f

public static DefaultFilterGroup ParseFilterList(string filters, DefaultFilterGroup defaultValue = DefaultFilterGroup.Default)

The output on MacOS with help default-filter-sets is now:

Duplicati has some built-in filters for different operating systems. In
addition to the sets below, there is the special set "All", which selects all
filters. These sets are specified via the --default-filters parameter.

    None: Selects no filters.
    All: Selects all the filter groups.
    Default: A set of default filters, currently evaluates to:
    SystemFiles,OperatingSystem,CacheFiles,TemporaryFolders,Applications.

    SystemFiles: Files that are owned by the system or not suited to be backed
    up. This includes any operating system reported protected files. Most
    users should at least apply these filters.
      *.dbfseventsd
      *.fseventsd
      /.com.apple.backupd.mvlist.plist
      /.fseventsd/
      /.hotfiles.btree
      /.MobileBackups
      /.MobileBackups.trash
      /.Spotlight-V100/
      /.TemporaryItems
      /.Trashes/
      /.vol/
      /afs/
      /automount
      /automount/
      /Backups.backupdb
      /cores/
      /Desktop DB
      /Desktop DF
      /dev/
      /home/
      /Library/Caches/
      /Library/Logs/
      /Library/Updates/
      /MobileBackups.trash
      /net/
      /Network/
      /Previous Systems
      /private/
      /System/Library/Caches/
      /System/Library/Extensions/Caches
      /tmp/
      /Users/Guest/
      /Users/Shared/SC Info/
      /Volumes/

    OperatingSystem: Files that belong to the operating system. These files
    are restored when the operating system is re-installed.
      */Library/Logs/
      /mach.sym
      /mach_kernel
      /Network/
      /Previous Systems*
      /sbin/
      /System/
      /Volumes/

    TemporaryFolders: Files and folders that are known to be storage of
    temporary data.
      *.Trash*
      */lost+found/
      */Microsoft User Data/Entourage Temp/
      */Network Trash Folder/
      */Trash/
      */VM Storage
      /tmp/
      /var/

    CacheFiles: Files and folders that are known cache locations for the
    operating system and various applications
      *.hotfiles.btree*
      *.Spotlight-*/
      */Application Support/Google/Chrome/Default/Cookies
      */Application Support/Google/Chrome/Default/Cookies-journal
      */backups.backupdb/
      */Google/Chrome/*cache*
      */Google/Chrome/Safe Browsing*
      */iP* Software Updates/
      */iPhoto Library/iPod Photo Cache*
      */iPhoto Library/iPod Photo Cache/
      */iTunes/Album Artwork/Cache/
      */Library/Application Support/SyncServices/
      */Library/Caches/
      */Library/Calendars/*/Info.plist
      */Library/Calendars/Calendar Cache
      */Library/Cookies/com.apple.appstore.plist
      */Library/Cookies/Cookies.binarycookies
      */Library/Mail/*/Info.plist
      */Library/Mail/AvailableFeeds/
      */Library/Mail/Envelope Index
      */Library/Mirrors/
      */Library/PubSub/Database/
      */Library/PubSub/Downloads/
      */Library/PubSub/Feeds/
      */Library/Safari/HistoryIndex.sk
      */Library/Safari/Icons.db
      */Library/Safari/WebpageIcons.db
      */Library/Saved Application State/
      */Mozilla/Firefox/*cache*
      *MobileBackups/
      /Desktop DB
      /Desktop DF
      /System/Library/Extensions/Caches/
      [.*/(cookies|permissions).sqllite(-.{3})?]

    Applications: Installed programs and their libraries, but not their
    settings.
      /Applications/
      /Library/
      /opt/
      /usr/

And the short version (shown under Advanced options in the UI):

 --default-filters (String): Default filter sets
    This option specifies which built-in filter sets that should be applied.
    Multiple filters can be supplied, separated with commas.
    None:
    Selects no filters.
    All: Selects all the filter groups.
    Default:
    A set of default filters, currently evaluates to:
    SystemFiles,OperatingSystem,CacheFiles,TemporaryFolders,Applications.
    SystemFiles: Files that are owned by the system or not suited to be
    backed up. This includes any operating system reported protected files.
    Most users should at least apply these filters.
    OperatingSystem:
    Files that belong to the operating system. These files are restored when
    the operating system is re-installed.
    TemporaryFolders: Files and
    folders that are known to be storage of temporary data.
    CacheFiles:
    Files and folders that are known cache locations for the operating system
    and various applications
    Applications: Installed programs and their
    libraries, but not their settings.
    * aliases: --default-filter
    * default value: Default

Lots of details to consider, but for now I have made All = Default. The division between Cache and Temporary is a bit hard, but I have used the idea that Temporary are folders or files that the system uses/exposes/names and Cache is stuff that applications are known to write.

It should be easy to add a dropdown of the kind that @renestach proposes.

[tygill]

I just took a look at this change, and it seems like a pretty good start to me so far, (though having someone else review to make sure the paths seem to fit the groups they've been assigned to would be good). I left a few comments inline in the diff, mostly with a few questions about whether certain paths should be included / are in the right group.

[kenkendk]

I am sure we can discuss what goes where for a long time, and that is fine. I have updated the branch with @tygill 's suggestions.

One thing I am not entirely sure of yet, is the log files. A common user would not want these files backed up, they would want "documents and settings" only, but a sysadmin would certainly want these included. Does it make sense to have a "log files" group as well?

[tygill]

I just took @kenkendk 's filter_group branch and did some more work with it:
https://github.com/tygill/duplicati/tree/feature/filter-groups

The main thing is that I've implemented the earlier suggestion to use filters via the --exclude={groupName} syntax. (And I've removed the --default-filters parameter and anything related to it.) I've also cleaned up some things related to Windows filters.

One larger impacting change I've made is removing the All group, and renaming Default to DefaultExcludes (with a corresponding DefaultIncludes that can have things added in in the future). The main reason for this is that, with support for using filter groups with the --include and --exclude parameters, if we want to add any filter groups that should be included, using --exclude={all} might inadvertently cause things that should be backed up to be missed, which seems like something we should avoid.

The main thing that I know is still missing is the UI. I haven't yet removed the original default filters UI, or replaced it with support for filter groups using the new syntax. I'm also not sure if there are parts of Duplicati that will have problems with filters that look like {group} (e.g., the database where UI settings and filters are stored) instead of [regex] or simple.

But hopefully this gets this a bit closer to being officially revived :)

[tygill]

I just removed the default filters UI on my branch, and added an option for excluding filter groups to the regular filter drop down (though it doesn't prompt you with any list of available groups or anything - I tried to look at hiding the text entry when the feature group option was selected, but I don't know enough about Angular to figure out how to hook that up.)

[tygill]

I took another look at the UI and was able to get a drop down with available filter groups added in place of the text box, so you can now add filter groups without needing to know their names! (It's far from perfect, and it won't recognize / parse filter group aliases like OS or System if you enter them manually in the list editor, but it at least works if you don't try anything like that.)

I think now the only major thing brought up in this thread that isn't included is displaying information about exactly what each group excludes in the UI. I know doing that is possible, but I don't think I'm familiar enough with AngularJS or Duplicati's web interfacing to hook that up anytime soon. Is there anyone else that could take a look at that soon, or should I submit a pull request with what I have for now?

[Pectojin]

I like the inline dropdown as it makes it fairly easy to see what options there are and it fits with the existing UI.

Would it make sense to add the description text once you select an item in the dropdown? Sort of in the same way that advanced options provide descriptions after the option was selected?

[tygill]

I think that would make sense, and hard coding the description text should be pretty straight forward (though there some things with how the filter strings are bound to the model and parsed back out that I wasn't able to understand, so I'm not sure I know how to add a new field in easily).

The part I'm really not sure how to do would be adding the full list of filter rules, since that would like require adding a new web endpoint to request the list of rules for a given group.

[Pectojin]

Right now it seems that inlining the hints is the standard. /Duplicati/Server/webroot/ngax/templates/addoredit.html#L331

At least that's what was done for the Backup retention dropdown on the last page. I don't see any reason we couldn't do the same here.

I know that the advanced options are entirely dynamically loaded from the backend, but they're arguably much more dynamic and changes more often than these options would.

Edit: I think I didn't read your reply to the end and ended up not answering your question... That would be a good argument for having an API endpoint to pull the paths from.

[tygill]

Yeah, the list of filter paths has enough dynamically generated stuff in it (pulling environment variables, operating system specifics, and system registry settings) that I don't think a static list would be very useful.

That said, I'm curious whether people feel like that's a required feature before adding these back in. The previous discussion made it sound like that was what people were thinking, but with the subdivided groups I'm not so sure myself

[tygill]

I'm marking this issue as resolved, since the initial change has now been merged. I've opened #3180 as a followup for exposing more details about the contents of the filter groups in the UI.

[Pectojin]

Sounds good. Thanks for your efforts on this @tygill 👍

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/only-exclude-file-filter-works/9807/7