Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error outeput from restic prune #2642

Open
daffyDukk opened this issue Mar 11, 2020 · 5 comments
Open

Error outeput from restic prune #2642

daffyDukk opened this issue Mar 11, 2020 · 5 comments

Comments

@daffyDukk
Copy link

@daffyDukk daffyDukk commented Mar 11, 2020

Output of restic version

restic 0.9.6 compiled with go1.13.4 on linux/amd64

Describe the issue

Restic is running on a Synology DSM 918+ doing offsite backup to Wasabi

I get the following error when running restic prune

repository 27b7f3ca opened successfully, password is correct
counting files in repo
building new index for repo
[14:58] 100.00%  87026 / 87026 packs
repository contains 87026 packs (433060 blobs) with 406.435 GiB
processed 433060 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 44 snapshots
[0:24] 100.00%  44 / 44 snapshots
Fatal: number of used blobs is larger than number of available blobs!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new

So, reporting the error. If you need more info dont hesitate to ask

@dimejo

This comment has been minimized.

Copy link
Contributor

@dimejo dimejo commented Mar 11, 2020

Please see #2045

@daffyDukk

This comment has been minimized.

Copy link
Author

@daffyDukk daffyDukk commented Mar 11, 2020

ok. have looked there. Could not find a solution or explanation as to the error.

@MichaelEischer

This comment has been minimized.

Copy link

@MichaelEischer MichaelEischer commented Mar 11, 2020

What output does restic check generate (please store a full copy of the output somewhere, in case the full output might become relevant later on)? I'm especially interested in whether it complains about "pack <...> not referenced in any index" or "pack <...> does not exist". Does it report any missing trees or blobs?

Does the error still show up if you start prune a second time?

How often do you run the check and prune commands? Did something unusual happen before the prune error showed up, like an interrupted / crashed backup?

The error usually indicates that some data blobs were lost. If the prune error persists on a second run, you should run restic rebuild-index (but please run restic check before and keep a copy of the full output) as soon as possible to rebuild the index and ensure that it does not refer to missing data blobs.

@daffyDukk

This comment has been minimized.

Copy link
Author

@daffyDukk daffyDukk commented Mar 14, 2020

Hi. Sorry for the late reply.

I run restic check twice a day and prune once a day. I cannot see a crash or failed backup in the logs.

Se attached files for output from commands ran in following order:
1: Restic check
2: Restic prune
3: Restic prune
4: Restic rebuild-index
5: restic prune
01check_before_output.txt
02prune_after_first_check.txt
03second_prune_after_first_check.txt
04rebuild_index.txt
05prune_after_rebuild_index.txt

@MichaelEischer

This comment has been minimized.

Copy link

@MichaelEischer MichaelEischer commented Mar 22, 2020

Hmm, something strange is happening here:

  • The check run does not complain about not referenced or missing packs, which means that the index contained every pack. But still some blobs were missing. Thus some data is missing from the repository.
  • The next two prune runs fail, but until then their results are identical.
  • rebuild-index seems to be ok.
  • The last prune run should just yield the same results as before. However, it is for some reason three times slower and reports errors for packs that seemed to be fine before.

Can you manually download one or two of the "incomplete pack files" from wasabi and check whether their sha256 hash matches (shasum -a 256 <filename>)? They should be store with a filename like data/<first two characters>/<hash>, e.g. data/db/db51b607eee8d3a804f90519b568f4543050e5b8e24b10a4f5e77623b9cd9776.

As it looks like you are still backing up into that repository, please follow these steps to make sure that new backups are (quite likely) not affected by these errors:

  • Run rebuild-index (which you already did) to make sure that the index only references chunk that actually exist.
  • Run backup -f <path...> to have restic fully rescan all files and upload any missing chunks. The important part here is the -f (=--force) flag.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.