Corrupted Repositories & Can't Recover

[benlilley]

Output of restic version

restic 0.9.3 compiled with go1.11.1 on linux/amd64

Describe the issue

We have some what appear to be very broken repositories, we're not sure what's happened, but we're trying to work how to recover them. We have probably a 100 or so in this state, but equally have another 100 that are fine. When running a restic check we get output like this, note this is after running a restic rebuild-index:

using temporary cache in /tmp/restic-check-cache-329227276
repository a6220f6c opened successfully, password is correct
created new cache in /tmp/restic-check-cache-329227276
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree a6e44d4d:
  tree a6e44d4d: file "Screen-Shot-2018-05-29-at-9.56.34-AM.png" blob 0 size could not be found
  tree a6e44d4d: file "TAU-Water-40.jpg" blob 0 size could not be found
  tree a6e44d4d: file "Team-Shots-5-of-5.jpg" blob 0 size could not be found
  tree a6e44d4d: file "Traffic-Management-banner.jpg" blob 0 size could not be found
  tree a6e44d4d, blob 6e784d27: not found in index
  tree a6e44d4d, blob 7da1f409: not found in index
  tree a6e44d4d, blob 233c2345: not found in index
  tree a6e44d4d, blob 7855ce5a: not found in index
error for tree cdcf5732:
  tree cdcf5732: file "Screen-Shot-2018-05-29-at-9.56.34-AM.png" blob 0 size could not be found
  tree cdcf5732, blob 6e784d27: not found in index
error for tree 6c7f34d3:
  tree 6c7f34d3: file "Easter-eggs.PNG" blob 3 size could not be found
  tree 6c7f34d3: file "Easter-eggs.PNG" blob 4 size could not be found
  tree 6c7f34d3: file "Easter-eggs.PNG" blob 6 size could not be found
  tree 6c7f34d3: file "IMG-5311.JPG" blob 1 size could not be found
  tree 6c7f34d3: file "IMG-5313.JPG" blob 2 size could not be found
  tree 6c7f34d3, blob 5b0ea985: not found in index
  tree 6c7f34d3, blob 28362a7d: not found in index
  tree 6c7f34d3, blob 88da5cdc: not found in index
  tree 6c7f34d3, blob 2baafaeb: not found in index
  tree 6c7f34d3, blob c0fdfaab: not found in index
error for tree e8999dc6:
  tree e8999dc6: file "Screen-Shot-2018-05-29-at-9.56.34-AM.png" blob 0 size could not be found
  tree e8999dc6: file "TAU-Water-40.jpg" blob 0 size could not be found
  tree e8999dc6: file "Team-Shots-5-of-5.jpg" blob 0 size could not be found
  tree e8999dc6: file "Traffic-Management-banner.jpg" blob 0 size could not be found
  tree e8999dc6, blob 6e784d27: not found in index
  tree e8999dc6, blob 7da1f409: not found in index
  tree e8999dc6, blob 233c2345: not found in index
  tree e8999dc6, blob e2c35441: not found in index
  tree e8999dc6, blob 7855ce5a: not found in index
error for tree 4f77bc5c:
  tree 4f77bc5c: file "Picture-1.jpeg" blob 0 size could not be found
  tree 4f77bc5c, blob e2c35441: not found in index
error for tree 7adae8d6:
  tree 7adae8d6: file "Screen-Shot-2018-05-29-at-9.56.34-AM.png" blob 0 size could not be found
  tree 7adae8d6, blob 6e784d27: not found in index
error for tree f9d48fe5:
  tree f9d48fe5: file "corporate-banner.jpg" blob 0 size could not be found
  tree f9d48fe5, blob 247de84f: not found in index
error for tree 4d4591ba:
  tree 4d4591ba: file "labourers-banner.jpg" blob 0 size could not be found
  tree 4d4591ba, blob 2ca4a1bf: not found in index
error for tree f195043b:
  tree f195043b: file "Traffic-Management-banner.jpg" blob 0 size could not be found
  tree f195043b, blob d7297bd9: not found in index

As this was after rebuilding the index we also tried a restic prune which resulted in this:

restic prune
repository a6220f6c opened successfully, password is correct
counting files in repo
building new index for repo
[1:30] 100.00%  7948 / 7948 packs
repository contains 7948 packs (375700 blobs) with 33.587 GiB
processed 375700 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 194 snapshots
[0:23] 100.00%  194 / 194 snapshots
Fatal: number of used blobs is larger than number of available blobs!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new

Some advice on how to proceed from here would be appreciated, and thanks for the work you're doing on this project.

[dimejo]

Which backend are you using? Did you interact with the repo with anything else than restic?

[benlilley]

We're using a Ceph cluster as the storage backend via the S3 API. And no nothing else is touching the repository.

[dimejo]

Looks like a duplicate of #2045.

[fd0]

Sorry for not getting back to you earlier.

It looks like some files which were written to the backend earlier are missing now. To get out of this situation, you need to manually remove the snapshots which reference objects that are not available any more. I'm sorry that this is a manual process for now. Use restic find --blob <blob id> to get the snapshot which references it, then use restic forget <snapshot id> to remove it.

I'm tentatively closing this as a duplicate of #2045, but please do add further comments (and report back if you were able to recover!). Thanks!