Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
restic takes 10-15 minutes to do anything (eg list snaphots) #1959
I installed restic a couple of weeks ago on my NAS (ReadyNas 104 -- installed by compiling go 1.4 with gcc, then using go 1.4 to compile go 1.10.3, and finally using go 1.10.3 to compile restic).
I've been uploading data to Backblaze B2 successfully, with a few hundred GB uploaded over the last fortnight. (Aside: I haven't checked if that's fast or slow but it was running non-stop and continuously showed updates in the console).
In that final upload, I saw a lot of errors like this:
My actual problem
After the last upload, yesterday I ran
I've compiled a debug binary and generated the following (redacted) log for
I can reproduce this easily. I imagine it's not going to be easy for anybody else since I'm just running vanilla commands! I'm hoping there will be something in those logs which indicate what's changed in my installation or environment, or perhaps in my backup repo, that has made this slow down so much. I'm happy to run this again with different flags, or to drill down through the file system on B2 to look for things -- whatever will help.
Thanks in advance! I'm looking forward to getting this up and running fully.
Hm, interesting! restic spends the 15 minutes checking lock files in the repo. I'm guessing you're located in Europe, so each operation which needs a response from the B2 API will have a very high latency. That's a common problem with B2.
So, for each of the 618(!!!) lock files in the repo, restic issues a download request, parses the lock file, then moves on to the next lock file. That's what takes so much time!
You can remove stale lock files by calling
Is it maybe possible that you're using credentials for accessing B2 which don't have the permission to remove files? It looks like restic's lock files stay in the repo, and since a lock file is replaced every five minutes or so, that left many files there...
Thanks for that diagnosis. So the failure to delete lock files was relevant after all! I didn't see that message the first few times I ran restic, nor did I see it for the first half of the days-long upload. I've run
I authenticate with my master account details (with the intention of switching to app-specific keys shortly), so I would hope it's not permissions problems, assuming everything is working correctly at B2.
I was away for a week, and left restic doing its thing. While I was gone my Internet connection died for some number of days. Could that have accounted for the lock files still being there? If so, is there anything I can do to guard against this in future? Since I've got just the one box backing up, I'm thinking I could run
Thanks for your help!
My understanding is that
I might be wrong, and I don't know the answers to your first questions. I'm also curious how
Cool, thanks for the feedback!
No, I don't think that was the issue of restic not being able to delete lock files. Sometimes B2 is a bit unstable, maybe that was one of the reasons. In your debug log, restic managed to delete the lock file it created just fine (right at the end of the log).
Apart from using
@mholt is indeed correct: By default,
Ideally, we come up with an algorithm which removes the need for lock files.
I'm going to close this issue for now, let us know if you see the error messages again (so then we can maybe debug it). Please feel free to add further comments. Thanks!
@mholt, everyone; not wanting to be OOT nor 'necroposting' here, but just to serve as a warning for other folks who may come a-googling in the future (which is how I found this issue, and the advice below):
That backup took a few more hours than usual, and finally aborted without writing a snapshot, wasting almost a full day of running :-/