Error making mount request: failed to login: Error making API request

[eravindar12]

I am seeing the following issue and due to this unable to unseal the keys on one of the vault node.

2022-08-02T21:17:33.930Z [INFO] server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=3.701610433s grpc.code=Unknown
err=
| error making mount request: couldn't read secret "xxxx": Error making API request.
|
| URL: GET http://vault:8200/v1/xxxx/data/configuration/xxx_config
| Code: 503. Errors:
|
| * Vault is sealed

->$ vault operator unseal
Unseal Key (will be hidden):
Error unsealing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:

unable to retrieve stored keys: failed to decrypt keys from storage: cipher: message authentication failed

I tried to provide all of the 5 keys but it’s failing when I pass the 3 rd key every time I see the below error.

However, the unseal keys worked on one of the nodes vault-2 but failed on vault-1

[eravindar12]

It works after I cleaned all the files in /vault/data and restart the pod.

#cleanup old data

#kubectl exec -it vault-1 -n vault ash

#rm –rf /vault/data

#kubectl delete pod vault-1 -n vault

#when new pod started, join the cluster

kubectl exec -it vault-1 -n vault ash

#vault operator raft join http://vault-0.vault-internal:8200/

#vault operator unseal (input unseal keys)

#check node status

#vault operator raft list-peers