Vault will not automatically unseal even if using an autoseal

[Raghavendarlokineni]

Describe Bug
core: entering seal migration mode; Vault will not automatically unseal even if using an autoseal: from_barrier_type=shamir to_barrier_type=awskms

==> Vault server configuration:

       AWS KMS KeyID: *******
       AWS KMS Region: *******
       Seal Type: awskms
       Api Address: http://127.0.0.1:8200 
        Cgo: disabled
     Cluster Address: https://127.0.0.1:8201 
          Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
           Log Level: info
               Mlock: supported: true, enabled: false
             Storage: mssql
             Version: Vault v1.1.0
         Version Sha: 36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd

 ==> Vault server started! Log data will stream in below:

2019-06-03T09:09:50.654Z [WARN]  core: entering seal migration mode; Vault will not automatically unseal even if using an autoseal: from_barrier_type=shamir to_barrier_type=awskms`

To Reproduce
Steps to reproduce the behavior:

1. Run vault server -config=/etc/service/vault-server/config.hcl

Expected behavior
Vault should get unsealed when integrated with AWS-KMS.

Environment:

• Vault Server Version (retrieve with vault status):

vault status
Key                           Value
---                           -----
Recovery Seal Type            shamir
Initialized                   true
Sealed                        true
Total Recovery Shares         5
Threshold                     2
Unseal Progress               0/2
Unseal Nonce                  n/a
Seal Migration in Progress    true
Version                       1.1.0
HA Enabled                    false

• Vault CLI Version (retrieve with vault version):

sh-4.2$ vault version
Vault v1.1.0 ('36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd')

• Server Operating System/Architecture:

cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"

Vault server configuration file(s):

disable_mlock = true

storage "mssql" {
  server = "10.10.10.10"
  username = "VAULT"
  password = "*****"
  database = "VAULT"
  table = "VaultTable"
  appname = "VaultApp"
  schema = "dbo"
  connectionTimeout = 30
  logLevel = 0
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/etc/service/vault-server/vault-server.crt"
  tls_key_file = "/etc/service/vault-server/vault-server.key"
}

ui = true
max_lease_ttl = "7200h"
default_lease_ttl = "7200h"
api_addr = "http://127.0.0.1:8200"

seal "awskms" {
  region     = "*******"
  kms_key_id = "******"
}

[tyrannosaurus-becks]

Hi @Raghavendarlokineni ! Thanks for posting this issue!

I tested this extensively on Vault 1.1.0 and I was able to successfully unseal using AWSKMS after having first unsealed it using Shamir. I think we need to figure out what's different about how we're doing it.

I posted extensive instructions on the vault-tool forum here. Mainly the process was, first start Vault with the first config and unseal it to get the Shamir keys. Then stop Vault and restart it with the second config and vault operator unseal -migrate. I think in your case that should do it. It might also be helpful to run Vault on debug to get more logs.

Another thing to check through would be if the AWS key and secret you're using locally have proper access to the key ID shown and it's in the proper region.

[Raghavendarlokineni]

Thanks, @tyrannosaurus-becks for sharing the info. We tried the procedure that you have shared above. But was wondering what is the use case of integrating vault solution with AWS KMS for auto unseal as for the first time we need to unseal it manually.

Instead, we have developed automation scripts which check for initialization, if not initialize and unseal it. Run this script as cron to check if Vault seals, if so this will auto-unseal with scripts developed.

[djordje-petrovic]

Got the same issue as OP. Can you even use auto-unseal if you haven't unsealed manually the first time?
Is this a bug, or intended behavior? Vault v1.1.3

[tyrannosaurus-becks]

@djordje-petrovic it's an intended behavior that you'd need to unseal Vault manually the first time because the Shamir algorithm is used to generate the Vault master key that's used for all encryption.

[tyrannosaurus-becks]

@djordje-petrovic actually, I stand corrected! A colleague checked in with me and let me know you should be able to do that.

[djordje-petrovic]

@tyrannosaurus-becks Thanks for clearing that up. So any idea what could be causing this? My vault status output is the same as OP's with:

Recovery Seal Type shamir although it should be awskms as specified in config file and Seal Migration in Progress true

[bartdzkan]

It always defaults back to shamir. I've seen it say AWS only when first initialized with it.
Then when the vault server reboots, it goes back to shamir even though it is aws auto unseal.

It's just a label, you can always check the config to see what is being actually used.

[djordje-petrovic]

Yeah, you are right. Anyway my issue was caused by a human error. I was trying to unknowingly join the new nodes to the existing cluster which has a manual seal. After deploying a new consul cluster with a different join tag everything was up and running.

[albertoal]

We were also hit by this when following the steps outlined on https://www.vaultproject.io/docs/concepts/seal.html#seal-migration when migrating our seal to awskms

The way we solved it was to run vault operator unseal -migrate against one vault server while the others are offline (i.e. vault service not running). Once the primary has been migrated, the others came up fine when starting the service, and are using awskms seal

[jefferai]

@albertoal Until the migration is finalized the new data isn't persisted, and until that happens other nodes in the cluster can't read it. We don't have a way to invalidate other cluster members at the moment because the seal is pretty fundamental to getting a node into a running state. So this is expected.

Closing as OP is gone and the other questions have been solved.