Very long startups for backup with many backups (>2300) #4664
Comments
|
Unfortunately our documentation is a bit out of date, but I think the
Is there a reason for keeping so many backup versions? If not, you might want to consider setting a retention policy to keep the number of versions more manageable. |
|
Thank you for the response and the tip about using retention. I was thinking about this option, but only throughout writing this issue I discovered the link between startup time and number of backups (which I previously had only assumed). What kind of consistency is checked with this option? And would it be possible to only check the consistency of a random subset of the |
|
This basically checks that various tables in the database are consistent with each other, in case some previous operation was interrupted for some reason. An option to perform a partial check is possible. However, I think the better solution would be to improve the performance of the query so that we can perform the check every time without substantial penalty. Improving the database queries is already a known issue. |
|
A performance improvement would be great! As that is probably not trivial, would it be possible to add this checking operation to the progress bar, such that the user has some feedback about the progress on their backup? Currently the text just states "Sicherung wird gestartet" (Backup is being started): |
|
I created issue #4671 to add this step to the progress bar. Did the |
Yes, very much so! Thank you for the recommendation. Additionally I have created a light retention policy which is slowly reducing the number of backups. One last question before closing this issue: Is there a way to trigger |
|
Currently, I don't think there's a way to trigger this manually. |
|
Thanks for the answer and the help so far :)! Disabling the consistency check and reducing the number of backups are good workarounds! Looking forward to those performance improvements! |
|
This issue has been mentioned on Duplicati. There might be relevant details there: https://forum.duplicati.com/t/multiple-backup-sources-and-one-destination-what-can-go-wrong/13864/6 |
Environment info
Description
Thank you for creating this great software and releasing it as open source! I have been using it for quite some while and recommended it to friends and family!
I have a backup (source 37.90 GB, backend size 77.56 GB) which has over 2310 valid backups and takes between 2h to "startup". I apologize for not having a more precise time estimate, as I have never timed it (the backup runs regularly), but I have been observing this issue for many weeks (it also may vary with the power setting of my laptop cpu). With "startup" I mean that the progress bar shows no progress, duplicati uses around 15% of the CPU.
Other backups on the same system, of partially the same files and with the same backend take much, much less time (in the order of minutes) but they have 12 or fewer valid backups.
Steps to reproduce
This is just my best guess on how to reproduce this:
Backup takes hours to get started but still utilizes CPU.
Debug log
Logs from "live/explicit only" (thousands of those):
I noticed that these logs iterate
FilesetID(I am not quite sure what this means, but I assume this is roughly the number of backups). Based on some more of the time stamps I calculated that 92 increments inFilesetIDtook exactly 5 minutes. According to my calculation my 2312 backups would take 125min to iterate, which matches my above finding that the backup takes about 2h to actually start.After iterating
FilesetIDto 2319 the backup actually starts -> the progress bar starts to say "Dateien ermitteln" (engl. maybe: find files) and counts files and file size up. Similarly in the log something happens (again abbreviated):Thanks in advance and of course I would be happy to provide more information if needed.
The text was updated successfully, but these errors were encountered: