Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Locked files seems to cause corrupt backup #3594
I've configured a backup of our Atlassian Stash data. The backup failed every time with "One or more errors occurred". I clicked "Show log", but there were no log entries for "General", and "Remote" showed only successful uploads of data files. Re-running the backup without changes frequently caused error messages "Detected non-empty blockset with no associated blocks". Deleting and recreating the database failed. The only way to continue was to delete the database, delete the remote files, and start over.
Since the GUI log showed no entries, I configured file logging and was able to identify a possible reason for the issues. It all seems to start when Duplicati hits a file that Stash has locked (there are apparently several of them), which causes this:
and the backup terminates. The suggested resolution for non-empty blocksets with no associated blocks seems to be to repair or recover the local DB, so I did that:
Obviously something was in an invalid state from the terminated backup, which the repair job tried to fix, but it did not resolve the issue. To recover from this issue, I had to delete the local database, delete all files in the remote location, make sure that all potentially locked files were excluded/enable shadow copy, and restart from scratch.
Steps to reproduce
I can't reproduce this with ordinary locked files, but there are a lot of ways to lock. Do you have a recipe that someone can follow with readily available equipment? We're on that chase in the forum, which includes some inspection of the database to confirm or refute guesses. Details at Fatal error: Detected non-empty blocksets with no associated blocks. Is your start-from-scratch also re-breakable by locked files? That would be a start. Possibly you'd want to use a test backup, not your production one, but cutting this down seems key to a fix.
If you're technically inclined, my locking is share-access. I'm still looking for an easy way to do byte-range, or to tell which variety some app is using. My locked files make only metadata entries in the backup. It's the file data ones that cause the problem. There's also a chance it's a latent error that needs compact or something.
Can you help find exact simple steps? I'm pretty sure I tried your steps 1 and 2 in my testing, with no luck...
I wrote a simple console app to lock a file in every way possible (in .NET) and configured Duplicate to back up the files. All the different locks cause Duplicati to log an IOException (either the file is completely locked or parts are locked), but nothing makes the backup fail. So locking might not be the culprit after all, and it's just a coincidence that the failures stopped when I excluded the locked files.
I can provide the complete log file from the failed and successful backups (info log level) if that helps. I'll also start looking at other causes.
When I was deleting and recreating the backup job to start over after database corruption, and was altering the configuration to exclude locked files, I see that I at one point changed the FTP backup from FTP (Alternate) to FTP. I chose FTP (Alternate) in the first place because it was able to report success on "Test connection"; FTP was never able to complete a test. But this seems to have other effects as well. FTP (Alternate) is much worse at connection handling (e.g. #3592) and "Channel is retired" exceptions are frequent with FTP (Alternate) and not reported at all with FTP.
The coincidence might be that I both excluded locked files and change the backend to FTP, and that in reality it was the change to FTP that made the issues disappear.