ERROR: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: getCacheForContentID: BLOB not found #1087
Comments
|
Can you provide the following:
Ideally, if you can hop on https://slack.kopia.io, we can chat interactively and hopefully figure it out quickly. |
|
Current working theory is that this was due to multiple overlapping maintenance jobs that ran and messed up blob deletion. The environment was kubernetes cron jobs where each call was overriding the hostname, so regular file-based lock did not protect against it. We'll need to do more debugging. |
|
Without having additional information, that'd be my hypothesis as well. |
|
I too just have run into the same issue. kopia snapshot verify shows: listing manifest contents and when connecting from various clients, i see this: [2021-05-23 12:27:12.037] [info] failed to open repository: unable to create shared content manager: error loading indexes: error listing index blobs: error listing indexes: error listing m blobs: error listing blobs: error listing all blobs: error reading directory: unable to complete ReadDir:\192.168.2.3\backups\kopia/m98 despite 10 retries Really not sure what happened... as we speak, I cannot connect from any Windows CLI, just from a linux CLI that I have... |
|
Seems you're connecting to a repository mounted over UNC path, can you ping me on https://slack.kopia.io so we can diagnose further? |
|
I am on slack, and sent you a DM. let me know if you have time. |
|
We tried recovering all indexes in @xxxliqu1dxxx 's repository and that brought it to somewhat-recovered state, but several blob files were missing. Figuring out why and this happened wasn't possible without all logs. I think we need to ensure we have logs to understand this better in the future - I'm going to add a feature that low-level stores content manipulation logs in the repository itself for some amount of time, so we can get more information about what leads to situation like this. I suspect in no particular order a) time sync issues, b) incomplete data migration, c) kopia maintenance bugs, d) problems with the blob storage provider or the underlying filesystem. |
|
Been talking with Jarek pretty much the whole day on this... Additional information for documentation purposes. The repo sits on a SMB Share, accessed from Windows, but sitting on an Unraid backend. In addition, though all snapshots have always run "fine" prior to this strange issue and the repo now seems corrupted beyond repairs, when I initiated a kopia restore from the linux cli, running off a Unassigned Device in Unraid, the CLI was using /root as temporary directory to perform its things, and these messages showed up not even 30 seconds after initiating a restore: got error can't write temporary file: write /root/.cache/kopia/ee4ed598dcadfdc7/contents/e6/233c7f6a2e12fc1ce18754093c4b64.f.tmp.ac76a49f9981f281: no space left on device when PutBlobInPath:/root/.cache/kopia/ee4ed598dcadfdc7/contents/e6/233c7f6a2e12fc1ce18754093c4b64.f (#0), sleeping for 1s before retrying This effectively was crashing the server and had to reboot it. Between the reboots, not sure if something in the Unraid Array corrupted the Repo, but that's where I am at right now. |
|
I'm suddenly facing the same problem, getting this similar error message:
and also
Based on what I read in this issue and in some related kopia forums, I tried a couple of things, but without success. Infos:
Questions:
Timeline:Ran backup A couple of days later, ran another backup rebooted machine a couple of days later Same day, ran another backup which failed Ran verify Ran full maintainance Verified content |
|
I was able to fix the issue. Below is a sequence of steps I attempted to fix it: Created missing blob files (see errors above) in remote repository Ran This resulted in the new error message: While this didn't fix the issue, it provided a confirmation in the error message which blobs are being accessed. Then inspected the logfile for the command (see https://kopia.io/docs/advanced/logging/). The content looks like this: Since and it contained references to the missing files. Following ideas from https://kopia.io/docs/advanced/consistency/
I then decided to remove that index file from the remote repository then re-ran and it ran successfully. |
|
Thanks, @jens-f ! Those steps worked for me. In my case I'm using Google Drive via rclone. A possible contribution to the corruption in my case was that for one of the index files there were duplicate directories in GDrive. Apparently it allows two different directories called "0b1" with different contents as siblings. 🙃 
|
My repository seems to be in a corrupted state somehow. I've done nothing other than periodically run snapshot create.
If I do a
kopia content verify:What do I do next?
The text was updated successfully, but these errors were encountered: