corrupt repository? error connecting to repository [...] BLOB not found

[probablykasper]

Shortly after renaming my macOS boot drive, I got this error in KopiaUI:

Error: error connecting to repository: unable to list sources: unable to find manifest entries: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found.

Logs

09:38:08.357 unable to check for updates: unable to open update state file: open /Users/k/Library/Application Support/kopia/repository.config.update-info.json: no such file or directory
--
09:38:08.388 password for /Users/k/Library/Application Support/kopia/repository.config retrieved from OS keyring
09:38:08.390 generating new TLS certificate
09:38:08.656 adding alternative IP to certificate: 127.0.0.1
09:38:08.660 starting Users/k/.local/bin/rclone-1_60_1
09:38:09.418 detected webdav address: https://127.0.0.1:53263/
09:38:09.582 throttling limits from connection info {"limits":{}}
09:38:09.586 finished initial cache scan {"cache":"contents","duration":"2.787934ms","totalRetainedSize":60,"tooRecentBytes":0,"tooRecentCount":0,"maxSizeBytes":5242880000,"inUsePercent":0}
09:38:09.649 finished initial cache scan {"cache":"metadata","duration":"62.684891ms","totalRetainedSize":95279724,"tooRecentBytes":71033663,"tooRecentCount":61,"maxSizeBytes":5242880000,"inUsePercent":1}
09:38:09.667 finished initial cache scan {"cache":"index-blobs","duration":"17.742097ms","totalRetainedSize":315541878,"tooRecentBytes":151342378,"tooRecentCount":55,"maxSizeBytes":5242880000,"inUsePercent":6}
09:38:09.668 refreshing committed state because it's no longer valid (now 2023-05-01 09:38:09.668047 +0200 CEST, valid until 0001-01-01T00:00:00Z)
09:38:09.668 refreshAttemptLocked
09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":1}
09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":2}
09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":4}
09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":3}
09:38:13.661 [STORAGE] ListBlobs {"prefix":"xw","resultCount":1,"error":null,"duration":"3.993067599s"}
09:38:13.662 [STORAGE] ListBlobs {"prefix":"xe","resultCount":2,"error":null,"duration":"3.99385125s"}
09:38:16.559 [STORAGE] ListBlobs {"prefix":"xr","resultCount":5,"error":null,"duration":"6.891036144s"}
09:38:16.560 ranges: [{xr8_15_8bb3bf1ccc5e2369410e183ca75e812a-sa449d0540eacac8c-c1 10420663 2023-02-20 07:13:50 +0000 GMT} {xr0_7_9a9d2b0fb54b712a13d3b3681655c91e-sc802fceef3ed2710-c1 33320996 2022-12-14 02:14:20 +0000 GMT} {xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 60384988 2023-04-30 18:30:30 +0000 GMT} {xr0_21_c2d51b5cadac51c70a6248cbe3f3118f-s8acd5e95ac0ba410-c1 51911353 2023-04-09 00:08:12 +0000 GMT} {xr0_20_d4100dd696859736ab06aa4cecac0477-s26696249ab7724dd-c1 50161711 2023-04-04 13:42:52 +0000 GMT}]
09:38:28.244 [STORAGE] ListBlobs {"prefix":"xs","resultCount":30,"error":null,"duration":"18.576212592s"}
09:38:29.057 [STORAGE] ListBlobs {"prefix":"xn27_","resultCount":0,"error":null,"duration":"811.70718ms"}
09:38:31.988 [STORAGE] ListBlobs {"prefix":"xn26_","resultCount":10,"error":null,"duration":"3.742948669s"}
09:38:33.831 [STORAGE] ListBlobs {"prefix":"xn25_","resultCount":20,"error":null,"duration":"5.586130433s"}
09:38:33.831 current epoch 26, uncompacted epoch sets 20 10 0, valid until 2023-05-01T09:58:09.668173+02:00
09:38:33.831 not generating range checkpoint
09:38:33.831 adding incremental state for epochs 25..27 on top of [{xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 60384988 2023-04-30 18:30:30 +0000 GMT}]
09:38:33.831 Complete Index Set for [0..27]: [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1], deletion watermark 2023-02-06 22:56:31 +0100 CET
09:38:33.831 active indexes [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1] deletion watermark 2023-02-06 22:56:31 +0100 CET
09:38:33.832 use-indexes [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1]
09:38:33.833 combined index segments {"original":31,"merged":28}
09:38:33.833 expireUnused {"except":["xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1","xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1","xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1","xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1","xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1","xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1","xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1","xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1","xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1","xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1","xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1","xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1","xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1","xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1","xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1","xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1","xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1","xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1","xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1","xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1","xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1","xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1","xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1","xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1","xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1","xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1","xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1","xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1","xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1","xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1","xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1"],"minSweepAge":"1h0m0s"}
09:38:36.662 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"2.828023695s"}
09:38:36.662 getContentInfoReadLocked(m079d5af4425d88f0a5a62c45c0d4c415) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found
09:38:36.662 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"482.169µs"}
09:38:36.662 getContentInfoReadLocked(m3a8db689ff9d228135809c6d903999ad) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found
09:38:36.663 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"396.692µs"}
09:38:36.663 getContentInfoReadLocked(m47793d0d73bddc091df13f9f2610b1a1) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found
09:38:36.663 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"446.51µs"}
09:38:36.663 getContentInfoReadLocked(ma9934163cc21c036a0fbfbbe8ff636ae) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found
09:38:36.664 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"400.725µs"}
09:38:36.664 getContentInfoReadLocked(ma7d74bec69eb8e385dc13d9cbbf415fe) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found
09:38:36.664 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"395.983µs"}
09:38:36.664 getContentInfoReadLocked(me986cf021a6b3304548993a62e500273) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found

Then (probably after restarting KopiaUI), the repository doesn't show up in KopiaUI at all, it just shows the repository selection screen. If I try to reconnect to the repo, it goes to the Initializing repository... screen, then afterwards it just resets to the repo selection screen again without any error message.

Even though that screen shows up, I now see that I do have a repository.config file:

repository.config

{
  "storage": {
    "type": "rclone",
    "config": {
      "remotePath": "kbackup:kopia",
      "rcloneExe": "/Users/k/.local/bin/rclone-1_60_1",
      "atomicWrites": false,
      "dirShards": null
    }
  },
  "caching": {
    "cacheDirectory": "../../Caches/kopia/b9130b74a7154aba",
    "maxCacheSize": 5242880000,
    "maxMetadataCacheSize": 5242880000,
    "maxListCacheDuration": 30
  },
  "hostname": "k-imac",
  "username": "k",
  "description": "My Repository",
  "enableActions": false,
  "formatBlobCacheDuration": 900000000000
}

If I run /Applications/KopiaUI.app/Contents/Resources/server/kopia policy list, it gives me a similar error to the one I originally got:

ERROR error listing policies: unable to list manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found

[probablykasper]

In the meantime, is there any way to recover my policies?

[lupusA]

Hi @probablykasper,

can you ping me on Slack? If you just renamed your drive, then we might get your repository and other things back.

Cheers,

[faevourite]

Happened to me as well without renaming the boot drive. #1087 (comment)

[alexdrl]

Happened to me for the 3rd!!! time, any way to recover my repo and data?