Duplicati creates large files in the roaming part of the user profile on Windows #2222
Duplicati Version: 188.8.131.52_canary_2016-11-12
Windows roaming profile becomes large after longer usage of Duplicati. Login on computers that are part of a domain takes a long time due to the large profile. This is caused due to the files
Steps to reproduce
Actual result: Roaming profile increases in size, increased network traffic, long login delays
Since with this fix, sqlite files are saved in AppData\Local instead of AppData\Roaming, it will solve #1757 if user only ticks the box for "Application Data" and leaves "Local Application Data" unticked.
I think we need some gracefull handling of upgrades. If the user has checked
@FlohEinstein: Your fix only renames the variable? Is there a conflict with
For the original issue, the logic is here:
The logic should be something like:
Maybe we should also move the lock file that prevents two instances from running:
This is a bit harder to decide on, because we really want to "just change it", but on the other hand we would like upgrades to guarantee that there are not two instances running at the same time.
@kenkendk It's not a renaming of the variable, it's what variable is read from Windows. I first thought it would be enough. But that way, the AppData\Roaming folder would not be selectable for backup anymore. That's why I made a better version with comments.
Databases are saved into "Duplicati" in the location of where SpecialFolder.ApplicationData points to. While this might be alright for *nix, Windows has 3 different folders (I decided to only use the two that are usually used). When we change SpecialFolder.ApplicationData to point to "AppData\Local" instead of "AppData\Roaming", the databases, config and lock get saved to the right place, however the AppData\Roaming-content isn't backed up anymore. So I added another value to represent this folder.
Normal procedure for user would be to not backup "AppData\Local" since this among other things contains unnecessary stuff like the whole IE/Chrome/Firefox-cache. Thousands of files. Only "AppData\Roaming" contains the important stuff. Or at least it should.
I like the new version better, perhaps more confusing for the average user though?
If you look at the code that picks a database, it uses
You sure? It should...
If you want to receive the old folderpath in the new, you need to use
EDIT: Disregard that, you're right... I have to rethink this
OK, I think I got it.
Since we also have #1352 which is kind of the same problem on Synology, I suggest we address this as a larger project.
The server and GUI already use the environment variable
I think that could be applied to the
The trick is to make sure that existing users are not in for a surprise when they upgrade. Everything should preferably work right away.
I suggest that Duplicati looks in the
This is an interesting discussion.
Maybe it is even dangerous:
Moving the config to the local APPDATA folder fixes this, but some new questions arise about this:
The solution I use in my environment is to run Duplicati as a service with the SYSTEM account (which also makes using VSS possible, regardless of the user's permissions). But Duplicati should run all scheduled tasks in a standard setup to make it more reliable.
Solution could be to not use %APPDATA% or %LOCALAPPDATA%, but to store config and database in %PROGRAMDATA%. The %PROGRAMDATA% Environment variable points in most situations to C:\ProgramData, which is the same for all user profiles. This way all backup tasks are shared between all users that log on to a single computer. All users will see the same backup tasks and know which backup tasks will run on that PC.
There are some things to think about when using %PROGRAMDATA% for storage of Duplicati files.
If Duplicati runs with the SYSTEM account (for example when running it as a service, or Duplicati.Server.exe is executed from the Windows Task Scheduler using this account), the user profile folders point to C:\Windows\System32\config\systemprofile. This folder will not contain anything interesting for a backup, so if Duplicati runs with the SYSTEM account, the libraries to Documents, Pictures etc. should be hidden.
Third potential issue: what happens if User1 and User2 log on at the same time on a single PC? Will the web interface start on port 8300 and will each scheduled task run twice? This should be handled somehow.
Maybe the best option to fix all issues is to use the %PROGRAMDATA% location for storage of config files and install Duplicati as a service by default (MSI installer could ask for a startup mode: System Tray for current user or all users, as a service or no automatic startup, running as a service should be selected by default).
OK, now we're getting somewhere :-)
But keep the following in mind:
So the documentation needs to reflect that ANY user with access to the machine has access to nearly everything when he can access the WebGUI. Therefore: Password for WebGUI should be set.
But the idea of every user selecting their respective personal folders and adding them to the backup task is pleasant. As long as Duplicati is run as service by SYSTEM, it has the privileges to access the files of all the users (with the problems mentioned above). If Duplicati is run as application by a user, it might then not have access to all the source folders (e.g. when User has specifcally set ACLs to his folders)
Regarding the other questions:
Two users starting it as application:
If we keep the configuration in the AppData\Roaming and the local databases of the backup tasks in AppData\Local, and the user moves to another computer, taking Roaming with him, leaving Local and LocalLow. The task will fail at first, since there's no local DB. User has to run a database delete and repair.
If the paths to the Personal Stuff change, it gets interesting: Now I'm not completely sure about the backup task itself, but what I hope happens is this:
Regarding the main topic I propose the following, slightly varying from @kees-z 's idea:
That makes sense. I did not think about that, but this is a security problem that should be addressed somehow.
As far as I know, Duplicati starts the web interface on TCP port 8200 by default, if no port is specified. If port 8200 is in use, Duplicati will use port 8300. I assume it will continue testing ports 8400, 8500 and so on, until an unused port is found. So the same security problem applies here:
User1 starts Duplicati as an application, so this instance listens on port 8200.
I still have doubts about storing any config information in a user's APPDATA folder, it sounds quite dangerous to me. For the roaming folders (Documents, Pictures etc) this will work, but all other folders selected for backup could be deleted.
I finally got around to looking at this, and have implemented it in a backwards compatible way.
This ensures that any new installs will not have the old file, and will thus use the new location.
This is a less intrusive approach than moving the data (as suggested in #2304).
I have not addressed the "run as service" things mentioned here. I think they are better suited for #1739.