Kubernetes service registration vault-initialized label incorrect

[ribbybibby]

Describe the bug
It seems that when you start and unseal a vault server pod that has already been initialized, using kubernetes service registration, the vault-initialized label on the pod is not updated to reflect the actual initialization state.

$ kubectl get pod vault-2 -o 'jsonpath={.metadata.labels.vault-initialized}'
false
$ kubectl exec -it vault-2 -c vault -- vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true

From a brief look through the code it seems to me that the initialized state is initially set here as false:
https://github.com/hashicorp/vault/blob/master/command/server.go#L988

But the service registration is only notified here when initialization occurs:
https://github.com/hashicorp/vault/blob/master/vault/init.go#L400

If vault is already initialized then it seems the state is never updated to reflect that.

To Reproduce
Initialize and unseal a vault pod in Kubernetes with persistent storage and kubernetes service registration configured as shown below. Delete the pod and unseal it when it is recreated.

Observe the labels added to the pod.

Expected behavior
I would expect the value of thevault-initialized label to match the value returned by GET /v1/sys/init.

Environment:

• Vault Server Version (retrieve with vault status): 1.4.0
• Vault CLI Version (retrieve with vault version): Vault v1.4.0
• Server Operating System/Architecture:
  • Kubernetes 1.17.3
  • Linux vault-1 4.19.106-coreos #1 SMP Wed Feb 26 21:43:18 -00 2020 x86_64 Linux

Vault server configuration file(s):

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_key_file    = "/etc/tls/tls.key"
  tls_cert_file   = "/etc/tls/tls.crt"
}

storage "raft" {
  path    = "/vault/storage"
  node_id = "$(POD_NAME)"
}

telemetry {
  disable_hostname = true
  statsd_address = "localhost:9125"
}

service_registration "kubernetes" {
  namespace      = "$(POD_NAMESPACE)"
  pod_name       = "$(POD_NAME)"
}

api_addr      = "https://$(POD_NAME).vault.$(POD_NAMESPACE):8200"
cluster_addr  = "https://$(POD_NAME).vault-cluster.$(POD_NAMESPACE):8201"
disable_mlock = true

[tyrannosaurus-becks]

Hi! Ah, I see, so you're saying that the second time you bring up the pod, initialized remains false in service registration, but shows as true at the init endpoint. Thanks for bringing this to our attention and for providing such good steps to reproduce it!