Fatal: config cannot be loaded: ciphertext verification failed

[minhdanh]

Output of restic version

0.9.4

How did you run restic exactly?

restic snapshots
Fatal: config cannot be loaded: ciphertext verification failed

restic -vvv check --read-data
using temporary cache in /tmp/restic-check-cache-369159670
Fatal: config cannot be loaded: ciphertext verification failed

restic rebuild-index
Fatal: config cannot be loaded: ciphertext verification failed

What backend/server/service did you use to store the repository?

Minio

Expected behavior

restic lists all snapshots

Actual behavior

restic failed with the above error

Steps to reproduce the behavior

I have a cronjob to backup the files using restic. It used to work.
But after some days the job failed with the same error.
When I use the command line to check, all restic commands failed with the same error.

Do you have any idea what may have caused this?

No idea

Do you have an idea how to solve the issue?

No idea

Did restic help you or made you happy in any way?

Yes

[rawtaz]

Is there an SSL/TLS certificate involved that might have expired or for some other reason isn't trusted?

[minhdanh]

Ah, there isn't. I'm using a plain http service (a service of another namespace in Kubernetes) for minio with restic.
So maybe that shouldn't be the case.

/ # env | grep REP
RESTIC_REPOSITORY=s3:http://minio.minio.svc.cluster.local:9000/backup

[minhdanh]

The backups that rely on restic in our infra have been failing. Can someone please have a look?

[kun93]

I’ve had a similar issue in the past . I would recommend checking if you have multiple keys in your repo . You should be able to run ’restic ls keys’ to check or looking at your repo manually . If you do, then you can move the key with the newest creation date and then it should work . I don’t recall which restic version allows you to specify a key but you could try that flag as well

[minhdanh]

@kun93 That command also failed with the same reason:

restic ls keys
Fatal: config cannot be loaded: ciphertext verification failed

[rawtaz]

@minhdanh I'm at a loss as to why you're getting that error if the files in the repo are still intact. I'd suspect they've been corrupted somehow.

You could try running restic on the server where the repository resides, like restic -r /path/to/repo ls keys or whichever command you prefer, to take the transport out of the equation and verify that you can access the repo locally. If you can't do that, I'd suggest the config file is corrupted, if nothing else. But I could be wrong!

[minhdanh]

/export/backup/keys # ls -l
total 8
-rw-r--r--    1 root     root           449 Jun 21 09:32 7b5d7e94a5820e31b900b1a90629391b52d57c287f44634133243dbb917aa3f5
-rw-r--r--    1 root     root           450 Jun 21 08:46 af02ceef93fdb6e838b470509b9b6532df1420d4958ef62c57de6624e72bdc26

There're two files in this keys directory. Looks like somehow the repo was initialized twice

[rawtaz]

So move one of the keys out of that folder (keeping the file in another place), and see if that helps (as per @kun93's suggestion)? :)

[minhdanh]

Yes, I tried that already. And could run the commands to some extent:

/ # restic check
using temporary cache in /tmp/restic-check-cache-056873375
repository 9c292b88 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-056873375
create exclusive lock for repository
load indexes
error: error loading index 02a90db0: ciphertext verification failed
error: error loading index 0235af6e: ciphertext verification failed
error: error loading index 0143f146: ciphertext verification failed
error: error loading index 0332a838: ciphertext verification failed
error: error loading index 038336d4: ciphertext verification failed
...

[minhdanh]

So I tried to rebuild the index:

/ # restic rebuild-index
repository 9c292b88 opened successfully, password is correct
created new cache in /root/.cache/restic
counting files in repo
pack file cannot be listed 004b9801152f9705e93e455577145494c5acdfee277881aa8f9ea77582ef0545: ciphertext verification failed
pack file cannot be listed 008c5c17a5096e28b79ab04dcc7782ecf5202395f73648780772fcc8fe3e545d: ciphertext verification failed
pack file cannot be listed 0004246f58c72672e2fd03f6044cfc1e3d4d244c60a7287676b6cd2154899dc7: ciphertext verification failed
pack file cannot be listed 00514f780f81dc02f1253990f1a5dbb53d64fe3163da3be00560ec51a8abea76: ciphertext verification failed
pack file cannot be listed 00ad66dafe21c5b5ebaa0299089d50e201659a4397ded13b57cb092e48c61e17: ciphertext verification failed
pack file cannot be listed 00ed6031cab2e2a529e157e10ce8a7c281c5c9e40d70cba94b908d7130415200: ciphertext verification failed
pack file cannot be listed 0003a90674aa1ced3570b09fde9010e0faddc1627dc021c00dec75289301eb09: ciphertext verification failed
pack file cannot be listed 015532645087ce77c818ab586ce52cde4371bc84e0d2ac39ff163dc72447f611: ciphertext verification failed
pack file cannot be listed 0172db21076402bf56dcb4e9a1585dda811f68f76bc9c95e18b9b47a32428c3d: ciphertext verification failed
pack file cannot be listed 0188e1ce5730fe885d5f11b2a275d0e0df3ed79d0998be49f896abfeb681f132: ciphertext verification failed
pack file cannot be listed 0194b3907b244e3e3bcb0463d5c23dc1edf658658ad160048ec50154099e45bf: ciphertext verification failed
...
pack file cannot be listed ffff901020a5dc2465fb82b3c7718ab2be8d57c194f9c3f7327a921861d59571: ciphertext verification failed
[0:03] 100.00%  2065 / 2065 packs
finding old index files
saved new indexes as []
remove 761 old index files

[minhdanh]

Then run restic check again:

/ # restic check
using temporary cache in /tmp/restic-check-cache-426509527
repository 9c292b88 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-426509527
create exclusive lock for repository
load indexes
check all packs
pack e8ec1dba: not referenced in any index
pack 50cf3406: not referenced in any index
pack 29199e99: not referenced in any index
...
pack c76ba5d0: not referenced in any index
pack 131ca6fd: not referenced in any index
2065 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error: ciphertext verification failed
error: ciphertext verification failed
error: ciphertext verification failed
...
error: ciphertext verification failed
Fatal: repository contains errors

[minhdanh]

Then I run restic prune as suggested:

/ # restic prune
repository 9c292b88 opened successfully, password is correct
counting files in repo
building new index for repo
pack file cannot be listed 004b9801152f9705e93e455577145494c5acdfee277881aa8f9ea77582ef0545: ciphertext verification failed
pack file cannot be listed 008c5c17a5096e28b79ab04dcc7782ecf5202395f73648780772fcc8fe3e545d: ciphertext verification failed
pack file cannot be listed 0004246f58c72672e2fd03f6044cfc1e3d4d244c60a7287676b6cd2154899dc7: ciphertext verification failed
pack file cannot be listed 0003a90674aa1ced3570b09fde9010e0faddc1627dc021c00dec75289301eb09: ciphertext verification failed
pack file cannot be listed 00514f780f81dc02f1253990f1a5dbb53d64fe3163da3be00560ec51a8abea76: ciphertext verification failed
pack file cannot be listed 015532645087ce77c818ab586ce52cde4371bc84e0d2ac39ff163dc72447f611: ciphertext verification failed
...
pack file cannot be listed fff83f9d5e22c9524795e4114abb7ef08e88b1d3b60e6baccdd7d477792937e0: ciphertext verification failed
pack file cannot be listed ff1a73e2456b18d00983348a7e5630c37e0e04a34270176c16044ffb5318cbb9: ciphertext verification failed
[0:02] 100.00%  2065 / 2065 packs
repository contains 0 packs (0 blobs) with 0 B
processed 0 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
ciphertext verification failed
github.com/restic/restic/internal/crypto.init.ializers
        /restic/internal/crypto/crypto.go:30
runtime.main
        /usr/local/go/src/runtime/proc.go:188
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

[minhdanh]

Then restic snapshots:

/ # restic snapshots
repository 9c292b88 opened successfully, password is correct
could not load snapshot 0065ef5b: ciphertext verification failed
could not load snapshot 010d1718: ciphertext verification failed
could not load snapshot 01217efd: ciphertext verification failed
could not load snapshot 016845ec: ciphertext verification failed
...

So it looks like no snapshots is usable now.
What can I do?

[seqizz]

I just got the same error, after using Minio's mirroring feature. I think this is an issue of the storage.

Since my repo is relatively big (~1tb), mirroring took too much time and apparently it was not so successful (or there was a change on it for some reason while/after the operation).

I saw md5 hashes of the repository config file (/config) does not match between storages. After copying this file manually everything worked.

Not sure about your setup but please check if the storage is keeping everything as it is.

[rawtaz]

Worth mentioning is that people are seeing this type of error now and then, and it usually turns out to be memory or other hardware problems. It's worth investigating that. Do some hardware tests, etc.

[aawsome]

Mhh - maybe we should save config not as config but instead use the SHA256 hash as with all other file names?
Then a repository.List could give the right config name or give an errer if more than one config files exist. If there is a problem with corrupted config file, then the SHA256 check can point this out and there is no need to give this irritating "ciphertext verification failed" error...

I already pointed out about inconsequent handling of the config files, see #2498 and #2505 .

If wanted, I can prepare a PR to change the config file name which would be another way to solve #2498 and makes #2505 obsolete. However it would mean a change in the repo format...

[seqizz]

Hmm. I don't think this is needed (good idea but, yeah, changing the format).

In my case, root cause turned out to be this, which should have done this correctly. Even a basic rsync of the repository folder would be non-problematic.

Anyway I don't want to poison the original issue, just wanted to mention after seeing the same line ☮️

[seqizz]

Mhh - maybe we should save config not as config but instead use the SHA256 hash as with all other file names?
Then a repository.List could give the right config name or give an errer if more than one config files exist. If there is a problem with corrupted config file, then the SHA256 check can point this out and there is no need to give this irritating "ciphertext verification failed" error...

I already pointed out about inconsequent handling of the config files, see #2498 and #2505 .

If wanted, I can prepare a PR to change the config file name which would be another way to solve #2498 and makes #2505 obsolete. However it would mean a change in the repo format...

After chatting on the mc issue I've linked above, I think you were right.

config is the only file on the repository which doesn't have a hash-based name. Which causes issues since it doesn't have any differential size/name/ETag header (in case of s3 backend) if you re-create the repository. For s3 backend that creates a hashing need while copying a repository which is very ineffective in terms of traffic/read-write costs.

[penguinpowernz]

I'm also having this issue on one of our customers devices, except I got it the opposite way to OP. The config is decrypted fine, but it can't decrypt the snapshots and I just get a load of errors like this:

error: error loading index fa2e8f97: ciphertext verification failed

I found when I remove one of the keys (the first one in alphabetical order) then it complains instead about not being able to load the config:

Fatal: config cannot be loaded: ciphertext verification failed

So now I'm thinking I want to fix this so we can access the backups. Perhaps by simply re-encrypting the config file with the newer key and deleted the older one, but I don't think that's gonna be easy without hacking up a small go utility (my go is good by my cryptography is poor).

I'm not sure if this is the only customer unit affected, there's probably more.

[kun93]

I'm also having this issue on one of our customers devices, except I got it the opposite way to OP. The config is decrypted fine, but it can't decrypt the snapshots and I just get a load of errors like this:

error: error loading index fa2e8f97: ciphertext verification failed

I found when I remove one of the keys (the first one in alphabetical order) then it complains instead about not being able to load the config:

Fatal: config cannot be loaded: ciphertext verification failed

So now I'm thinking I want to fix this so we can access the backups. Perhaps by simply re-encrypting the config file with the newer key and deleted the older one, but I don't think that's gonna be easy without hacking up a small go utility (my go is good by my cryptography is poor).

I'm not sure if this is the only customer unit affected, there's probably more.

Its been a while but I believe Ive had similar issues in the past and I've had good luck doing a combination of the above with the original key plus using the branch below to fix the corrupt config as mentioned https://forum.restic.net/t/unable-to-open-config-file-can-i-restore-it/1648/26

I believe what happens is that the new key got far enough to create a new config and thats why you can't access the snapshots since it's not linked to the original key/config. Not sure if it helps you but figured I'd share just in case so you can dig in

[MichaelEischer]

I believe what happens is that the new key got far enough to create a new config

There's exactly one point in time at which restic writes the config file, and that is when calling init for a repository. So this can't be the reason for a damaged config file.

I've put together a branch with some less hacky code (and several guardrails) to replace a missing/broken config file. See https://forum.restic.net/t/fatal-config-cannot-be-loaded-ciphertext-verification-failed/3027/2 for a description and the link to the branch.

[MichaelEischer]

restic should check that a newly added / changed key actually works before deleting the old one. That way it's possible to avoid nasty surprises.

[fermulator]

ran into this today too -- created/init repo, then performed a backup
, then deleted the old app key in b2 backblaze, made a new one, updated it, tried to backup and:

+ restic --repo b2:myvault:myrepo backup --verbose --cache-dir .//.restic/.cache --cleanup-cache --exclude .restic --exclude .git .gitignore ./
open repository
Fatal: Fatal: config cannot be loaded: ciphertext verification failed

[aawsome]

ran into this today too -- created/init repo, then performed a backup
, then deleted the old app key in b2 backblaze, made a new one, updated it, tried to backup and:

Is this reproducible (e.g. with very small backup data)? If yes, can you send us the command to reproduce?

[MichaelEischer]

Closing this issue as there's nothing left to be done here, after #3429 was merged. If some of the problems mixed into this issue still occur, please open a new issue.