restic prune failing - tree XXX not found in repository #2700
Comments
|
restic is currently not able to find some of the directories referenced in one of the snapshots. It might be either just missing from the index or the data can be actually missing. The repository contains quite a lot of incomplete pack files, which looks like several of your backup runs got interrupted for some reason? Did you run Please post the output of |
|
I just ran restic find --treels -la |
|
Hmm, the size of the pack file seems somewhat reasonable. Can you run
As a next step it would probably be best to run |
|
I ran As for if other snapshots are failing, the above |
restic check |
|
Do you know whether some of your backup runs were interrupted by maybe network problems? 67 incomplete packs should not appear out of nowhere, unless this is a problem of the underlying storage, it should take several interrupted backup runs to accumulate that many incomplete packs. Could you run You could run |
|
Some runs have probably been interrupted. I've run daily backups using restic for ~2 years, and 67 interrupted backups does not surprise me. Yeah, I truncated the I tried looking up some random from the |
|
The Then to rule out the first two theories of what's happening:
The |
|
The Running |
restic rebuild-index -v |
restic checklog: restic_check.log |
|
The log for I currently don't see any obvious reason for the missing trees/blobs. You could start repairing the repository as a next step using |
|
I ran I'll go ahead an run |
|
Oh, sorry I somehow didn't see the log for the check run. Since This starts to look a bit like CIFS/SMB temporarily "forget" to list some of the pack files in the repository. At least that's the most reasonable explanation I have for why it helped to run Just as an explanation: Without missing data in the repository |
|
I've run |
|
Maybe trying to figure out what went wrong will be impossible, and the better approach is just to migrate the data. Does there exist a tool for migrating snapshots from this broken repo, to a new repo? |
|
The prune log is cut off. At least the part up to the incomplete line that warns about an incomplete file, does not show any errors. You could use #2606 to copy snapshots to a new repository. |
|
Very similar scenario with 0.9.6 but I had the opportunity to log all output of the various steps. 01-backup.txt I followed the above suggested procedure and still see corruption/non-found warnings. What would help? |
|
@drzraf Could you run Running the backup again and then removing old snapshots should also fix the errors you're seeing. For future reference, here's a summary of the important log parts:
So it looks like restic received the incomplete pack from the first failed upload attempt up to log |
|
Subsequent I'll redo the same step, but I wonder if something couldn't be improved about backup integrity out-of-the-box (or if there are issue already tracking that behavior)? |
Can you clarify what you mean? Out of the box restic produces backups that does have integrity and that you can verify the integrity of. |
This error message tells me two things: Firstly the affected pack file was fully uploaded to the storage backend (as prune would otherwise have failed during the initial reindexing) and secondly the file was somehow damaged. Can you download the file from The "hash does not match id" error was already discussed in great depth in #1999, and also in #1596 (and probably many more). There are also a few ideas to salvage blobs from damaged packs, e.g. #1727. I'm not aware right now of an issue to track adding end-to-end checksums for the backends (see #804 (comment) for some elaboration on that topic). For the download step there's #2302 with a suggestion to improve handling of bit-flips. [Edit]I've noticed that my interpretation of the error message is only fully accurate for the latest master branch. With restic 0.9.6 there is a chance that the initial rebuild-index check does not fail. However, if I understood you correctly then the |
Output of
restic versionrestic 0.9.6 compiled with go1.12.12 on linux/amd64How did you run restic exactly?
What backend/server/service did you use to store the repository?
Local filesystem (backed by CIFS)
Expected behavior
Prune not failing
Actual behavior
Prune failing
Steps to reproduce the behavior
Do you have any idea what may have caused this?
Likely some corruption in the repository. Not that I have any particular reasons to think it is corrupt.
Do you have an idea how to solve the issue?
No
Did restic help you today? Did it make you happy in any way?
Not today, but most other days :)
The text was updated successfully, but these errors were encountered: