Unseal failed, invalid key with third key - regression bug ?

[z00m1n]

Describe the bug
After upgrading to v.1.2.0, I can't seem to unseal the vault; the first two keys submitted seem to work, but the third fails with Unseal failed, invalid key. I also tried v1.1.5, same issue.

To Reproduce
Steps to reproduce the behavior:

1. Initialize vault:

$ json_data='{ "secret_shares": 5, "secret_threshold": 3 }'
$ vault_address='https://vault.mydomain.tld'
$ curl --data "${json_data}" --request PUT "${vault_address}/v1/sys/init"

2. Seal vault:

$ root_token='<not shown here>'
$ curl --header "X-Vault-Token: ${root_token}" "${vault_address}/v1/sys/seal"

3. Try to unseal vault, see error:

$ key_01='<not shown here>'
$ curl --data "{\"key\": \"${key_01}\"}" --request PUT "${vault_address}/v1/sys/unseal
# works
$ key_02='<not shown here>'
$ curl --data "{\"key\": \"${key_02}\"}" --request PUT "${vault_address}/v1/sys/unseal
# works
$ key_03='<not shown here>'
$ curl --data "{\"key\": \"${key_02}\"}" --request PUT "${vault_address}/v1/sys/unseal
# fails as described

Expected behavior
I expect the vault to unseal. If any keys are invalid, I would either expect all of them to be invalid or none of them: I initialize, seal and unseal using a number of shell scripts, have done so many times in e.g. the last six months, this used to work until a couple of weeks ago; the issue occurs for the first time today, with the new versions v1.1.5 and v.1.2.0. I am reasonably certain there are no copy & paste or other trivial errors involved.

Environment:

• Vault Server Version (retrieve with vault status):
  (also tried v.1.2.0)

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.1.5
HA Enabled         false

• Vault CLI Version (retrieve with vault version):

Vault v1.1.4 ('bbaaf165d317a1a64abc3c606a6d04793d571e91')

• Server Operating System/Architecture:
  • developing on macOS 10.14.6 (18G84)
  • deploying docker container with vault instance to Ubuntu 16.04.6 LTS

Vault server configuration file(s):

storage "file" {
  path = "/vault/file"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

api_addr = "http://0.0.0.0:8200"
ui = true
disable_mlock = true

Additional context
Installing packages to local macOS dev machine using brew; was apparently not updated yet to latest (client) version 1.1.5; hence client 1.1.4.

[z00m1n]

Tried going back to v1.1.4 to be able to continue to work, but there doesn't seem to be a docker image tag for that version anymore. Production is down.

[z00m1n]

Built a docker container for v1.1.4 by locally modifying hashicorp/docker-vault@b084fc5, replacing 1.1.3 by 1.1.4 (for which there apparently never was a docker image) instead of 1.2.0-beta1 (as the commit ref'd above did).

Used that locally built container as base for my own instance: Deployed the container using the same process, the same scripts as before; the issue persists.

Repeated the process for v1.1.3; issue persists. I'm pretty sure this was the version I used earlier. This indicated the issue in not caused by a new version of vault, but by something on my side.

Will have to dig deep; closing this for now.

[jefferai]

There isn't a Docker image for 1.1.4 because it was never released, and 1.1.5 just came out yesterday so the image hasn't been updated. See https://github.com/hashicorp/vault/blob/master/CHANGELOG.md#114115-july-25th30th-2019

[jeffgrunewald]

I can confirm seeing this exact issue with vault running 1.1.2

We have a file system storage backend and whenever we try to unseal the vault, even manually, the third of five attempted keys fails every time, regardless of which three keys are chosen.

Inputting any two keys successfully causes unseal to progress to 2/3. Applying any of the three remaining keys throws the "invalid key" failure and the unseal progress resets to 0/3.

If I then attempt vault operator unseal ${key} with the same key that previously failed moments before, it is immediately successful in progressing the unseal to 1/3.

[ncabatoff]

No keys are checked until we have enough to meet the required threshold. So you shouldn't read anything into the fact that you're able to submit the first two without error.

[jeffgrunewald]

Thanks; I was eventually able to work that out. Also resolved our issue although still unable to explain it.

For anyone who implements vault in a container on kubernetes with the file backend, it appears it is possible that when a pod transfers a persistent volume claim to the storage backend volume to a new pod, that claim maintains some sort of file lock from the old pod’s connection and the new pod is incapable of unsealing the vault, despite having access to valid unsealing keys. Deleting the claim along with the old pod allows the new pod full access to the volume and allows the new instance to correctly unseal itself.

[manoharnv]

We are using posgres as storage backend. even we are facing the same issue. Unseal fails. Any idea how to fix this?

We are using 1.2.0 version

[lz006]

when every third key fails (threshold is also set to three) then it might indicates you are using wrong keys somehow. We first thought that we are facing same issue than you folks. Although it turned out that our ansible which automates init and unsealing was misconfigured.

etcd 1.2.2 (docker image) and etcd 3.3.12