HA vault init with TLS - cannot validate certificate

[popopanda]

Hello,

I'm trying to setup HA vault cluster consisting of 3 vault pods in EKS.

I followed the TLS cert generation instructions from https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls/

When I try to run vault operator init, vault is returning:

Error initializing: Put https://127.0.0.1:8200/v1/sys/init: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

In my csr.conf, I have these defined:

[alt_names]
DNS.1 = vault
DNS.2 = vault.vault
DNS.3 = vault.vault.svc
DNS.4 = vault.vault.svc.cluster.local
IP.1 = 127.0.0.1

I also checked the csr generated:

            X509v3 Subject Alternative Name:
                DNS:vault, DNS:vault.vault, DNS:vault.vault.svc, DNS:vault.vault.svc.cluster.local, IP Address:127.0.0.1

I suspect 127.0.0.1 is from the env variable defined in the statefulset template

- name: VAULT_ADDR
value: "{{ include "vault.scheme" . }}://127.0.0.1:8200"
- name: VAULT_API_ADDR
value: "{{ include "vault.scheme" . }}://$(POD_IP):8200"

My tcp listener is configured as:

listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
  tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
  tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}

Is there another set of instructions I am missing?

Thanks

[woowil]

Here is my guideline how to create Vault TLS client and CA certificates

Variables

SERVICE=vault
NAMESPACE=vault
SECRET_NAME=vault-tls
CSR_NAME=vault-csr
BASENAME=vault
TMPDIR=./cert

Create a Certificate Signing Request (CSR)

# Create private key
openssl genrsa -out ${TMPDIR}/${BASENAME}.key rsa:2048 -days 3651

# Create a file ${TMPDIR}/${NAMESPACE}-csr.conf with the following contents
cat <<EOF >${TMPDIR}/${BASENAME}-csr.conf
[ req ]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = dn
req_extensions = v3_req
[ dn ]
C = NO
ST = Oslo
L = Oslo
O = Personal
emailAddress = youremail@domain.no
CN = ${SERVICE}.${NAMESPACE}.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE}
DNS.2 = ${SERVICE}.${NAMESPACE}
DNS.3 = ${SERVICE}.${NAMESPACE}.svc
DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
IP.1  = 127.0.0.1
EOF

# Create a CSR
openssl req -config ${TMPDIR}/${BASENAME}-csr.conf -new -key ${TMPDIR}/${BASENAME}.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/${BASENAME}.csr -days 3651

Create the certificate

# Create a file ${TMPDIR}/${BASENAME}.yaml with the following contents
cat <<EOF >${TMPDIR}/${BASENAME}-csr.yaml
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${CSR_NAME}
spec:
  groups:
  - system:authenticated
  request: $(cat ${TMPDIR}/${BASENAME}.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# Delete CSR and secret if exist
kubectl delete secret ${SERVICE_NAME} --namespace ${NAMESPACE}
kubectl delete csr ${CSR_NAME} --namespace ${NAMESPACE}

# Send the CSR to Kubernetes.
kubectl create -f ${TMPDIR}/${BASENAME}-csr.yaml --namespace ${NAMESPACE}

#If this process is automated, you may need to wait to ensure the CSR has been received and stored
kubectl get csr ${CSR_NAME} --namespace ${NAMESPACE}
## NAME        AGE     REQUESTOR      CONDITION
## vault-csr   2m44s   masterclient   Pending

# Approve the CSR in Kubernetes
kubectl certificate approve ${CSR_NAME} --namespace ${NAMESPACE}
## certificatesigningrequest.certificates.k8s.io/vault-csr approved

Store key, cert, and Kubernetes CA into Kubernetes secrets store

# Retrieve the certificate.
## If this process is automated, you may need to wait to ensure the certificate has been created. If it hasn't, this will return an empty string.
serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')

# Write the certificate out to the CRT file
echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/${BASENAME}.crt

# Retrieve Kubernetes CA
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > ${TMPDIR}/${BASENAME}.ca

# Delete secret (if needed)
#kubectl delete secret ${SERVICE_NAME} -all --namespace ${NAMESPACE}

# Store the key, cert, and Kubernetes CA into Kubernetes secrets.
### This will create files /vault/userconfig/vault.ca , /vault/userconfig/vault.crt, /vault/userconfig/vault.key in the vault Pods
### values.yaml: Update server.ha.config:listener, injection.cert:certName,certKey
kubectl create secret generic ${SECRET_NAME} \
    --namespace ${NAMESPACE} \
    --from-file=${BASENAME}.key=${TMPDIR}/${BASENAME}.key \
    --from-file=${BASENAME}.crt=${TMPDIR}/${BASENAME}.crt \
    --from-file=${BASENAME}.ca=${TMPDIR}/${BASENAME}.ca

# Verify the certificate:
openssl x509 -in ${TMPDIR}/${BASENAME}.crt -noout -text

Vault Helm Configuration (values.yaml)

# The below custom-values.yaml can be used to set up a single server Vault cluster using TLS. 
## This assumes that a Kubernetes secret exists with the server certificate, key and certificate authority:

global:
  enabled: true
  tlsDisable: false
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca # Matches the ${SECRET_NAME}/${BASENAME}.ca from above

injector:
  #certs:
   # secretName: vault-tls
    #caBundle: "**enter string base64 of vault.ca**"
   # certName: vault.crt
   # keyName: vault.key

server:
  image:
    repository: 'vault'
    tag: '1.3.4'

  extraVolumes:
  - type: secret
    name: vault-tls # Matches the ${SECRET_NAME} from above

  standalone:
    enabled: true
    config: |
      api_addr = "http://POD_IP:8200"
      listener "tcp" {
        address          = "0.0.0.0:8200"
        cluster_address  = "0.0.0.0:8201"

        tls_cert_file = "/vault/userconfig/vault-tls/vault.crt" # Matches the ${SECRET_NAME}/${BASENAME}.crt from above
        tls_key_file  = "/vault/userconfig/vault-tls/vault.key" # Matches the ${SECRET_NAME}/${BASENAME}.key from above
        tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca" # Matches the ${SECRET_NAME}/${BASENAME}.ca from above
      }

      storage "file" {
        path = "/vault/data"
      }

Verifications

# Checking certifiates on vailt Pods (webhook)
kubectl exec -it vault-0 -- ls /vault/userconfig/vault-tls
## vault.ca   vault.crt  vault.key

# Checking certifiates on agent-injecot (webhook)
kubectl exec -it vault-agent-injector-<POD_NAME_SUFFIX>-- ls /etc/webhook/certs
## vault.ca   vault.crt  vault.key

[popopanda]

I followed the instructions, and verified the certs all match from inside the pod. But still getting the same error.

[woowil]

I have removed injection.cert in my custom values file.
Is this related to missing Token Receiver resources?

$ kubectl get pod exampleapp-7fd844b549-4nkph**
NAME                          READY   STATUS     RESTARTS   AGE
exampleapp-7fd844b549-4nkph   0/2     Init:0/1   2          8h

$ kubecrt logs exampleapp-7fd844b549-4nkph -c vault-agent-init**
permission denied" backoff=2.969046366
2020-04-02T23:56:07.747Z [INFO]  auth.handler: authenticating
2020-04-02T23:56:07.791Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

[popopanda]

I don't believe so,

This is the only the vault-server pods during init phase.

When the pod first starts up, it logs

==> Vault server started! Log data will stream in below:

2020-04-03T21:00:26.353Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-04-03T21:00:31.530Z [INFO]  core: stored unseal keys supported, attempting fetch
2020-04-03T21:00:31.534Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2020-04-03T21:00:34.216Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-03T21:00:36.534Z [INFO]  core: stored unseal keys supported, attempting fetch
2020-04-03T21:00:36.543Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2020-04-03T21:00:37.235Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-03T21:00:40.233Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery

Then the vault pods will eventually crash loop, which I believe is normal behavior until the vault operator init is successful

[popopanda]

Interestingly, I edited the statefulset template env var for the vault-server from:

            - name: VAULT_ADDR
              value: "{{ include "vault.scheme" . }}://127.0.0.1:8200"

to

            - name: VAULT_ADDR
              value: "{{ include "vault.scheme" . }}://vault:8200"

But then I got the error:
Error initializing: Put https://vault:8200/v1/sys/init: x509: certificate is valid for vault.vault.svc, not vault

So I changed it to

            - name: VAULT_ADDR
              value: "{{ include "vault.scheme" . }}://vault.vault.svc:8200"

After that I was able to init the vault servers. Even though my altnames haven't changed on the csr.conf

[alt_names]
DNS.1 = vault
DNS.2 = vault.vault
DNS.3 = vault.vault.svc
DNS.4 = vault.vault.svc.cluster.local
IP.1 = 127.0.0.1

[woowil]

Moved post to #244

[woowil]

We got permission denied on the https request while patching (agent injection). The way to solve this
was to include DNS names for the service resource defined in the file (templates\injector-service.yaml).

The reason why vault.vault.svc worked in the StatefulSet env is probably because of CN (Common Name) definition in the CSR. The best option is not to hard code and change it back as before: "127.0.0.1:8200" or else vault Pods fail to install in other Kubernetes context/namespaces.

The TLS Certificate Signature Request needs to be something like this when using OpenSSL

Greetings from Norway

CSR file

cat <<EOF > ./certs/vault-csr.conf
[ req ]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[ dn ]
C = NO
ST = Oslo
L = Oslo
O = Our Business
OU = Our Department
emailAddress = email@domain.no
CN = vault.vault.svc

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = vault
DNS.2 = vault.vault
DNS.3 = vault.vault.svc
DNS.4 = vault.vault.svc.cluster.local
DNS.5 = vault-agent-injector-svc
DNS.6 = vault-agent-injector-svc.vault
DNS.7 = vault-agent-injector-svc.vault.svc
DNS.8 = vault-agent-injector-svc.vault.svc.cluster.local
IP.1  = 127.0.0.1
EOF

[woowil]

Hello,
I'm trying to setup HA vault cluster consisting of 3 vault pods in EKS.
I followed the TLS cert generation instructions from https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls/
When I try to run vault operator init, vault is returning:
Error initializing: Put https://127.0.0.1:8200/v1/sys/init: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
In my csr.conf, I have these defined:
[alt_names]
DNS.1 = vault
DNS.2 = vault.vault
DNS.3 = vault.vault.svc
DNS.4 = vault.vault.svc.cluster.local
IP.1 = 127.0.0.1

I also checked the csr generated:
X509v3 Subject Alternative Name:
DNS:vault, DNS:vault.vault, DNS:vault.vault.svc, DNS:vault.vault.svc.cluster.local, IP Address:127.0.0.1

I suspect 127.0.0.1 is from the env variable defined in the statefulset template

• name: VAULT_ADDR
  value: "{{ include "vault.scheme" . }}://127.0.0.1:8200"
• name: VAULT_API_ADDR
  value: "{{ include "vault.scheme" . }}://$(POD_IP):8200"

My tcp listener is configured as:
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}

Is there another set of instructions I am missing?
Thanks

Here is my config info.
Note the address, the cluster_address and tls_disable

   api_addr = "http://POD_IP:8200"

      # https://www.vaultproject.io/docs/configuration/listener/tcp/
      listener "tcp" {
        address          = "0.0.0.0:8200"
        cluster_address  = "0.0.0.0:8201"
        
        http_idle_timeout         = "5m"
        http_read_header_timeout  = "10s"
        http_read_timeout         = "30s"
        http_write_timeout        = "0"
        max_request_size          = 33554432
        max_request_duration      = "90s"

        # https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls/
        tls_disable        = "false"
        tls_cert_file      = "/vault/userconfig/vault-tls/vault-server-public.crt"
        tls_key_file       = "/vault/userconfig/vault-tls/vault-server-private.key"
        tls_client_ca_file = "/vault/userconfig/vault-tls/vault-client.ca"
        tls_min_version    = "tls12"
      }

[ikarlashov]

Just faced this issue today. You need to generate csr with the following config:

[alt_names]
DNS.1 = *.${VAULT_INTERNAL_SVC}
DNS.2 = *.${NAMESPACE}.svc.cluster.local
DNS.3 = *.${VAULT_INTERNAL_SVC}.${NAMESPACE}.svc.cluster.local

where

VAULT_RELEASE_NAME="vault"
VAULT_INTERNAL_SVC="${VAULT_RELEASE_NAME}-internal"

Then you can do:

vault operator init -address https://vault-0.vault-internal.vault.svc.cluster.local:8200

That's it :)

[j-sokol]

Using wildcard certificate as @ikarlashov noted worked for me.

One thing to mention, when joining the cluster from vault-1 and vault-2 pods, key, cert and ca-cert have to be provided:

vault operator raft join -leader-ca-cert="@${VAULT_CACERT}" -leader-client-cert="@${VAULT_TLSCERT}" -leader-client-key="@${VAULT_TLSKEY}" https://vault-0.vault-internal:8200'

where env variables above are set in Helm chart's values:

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca 
    VAULT_TLSCERT: /vault/userconfig/vault-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-tls/vault.key
  ha:
    enabled: true
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"

          tls_cert_file = "/vault/userconfig/vault-tls/vault.crt" 
          tls_key_file  = "/vault/userconfig/vault-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
        }

Also, certificates have to be configured in server.ha.raft.config.

[DennisLoska]

@j-sokol Thank you so much for the tip that certificates need to be provided this way, worked for me!!

In case anyone like me stumbles across this feel free to use my whole config, which works with the fixes described in this thread:

global:
  enabled: true
  tlsDisable: false

injector:
  enabled: false

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault/vault.key

  extraVolumes:
    - type: secret
      name: vault

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false

      config: |
        ui = true
        api_addr = "http://POD_IP:8200"

        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"

          tls_cert_file = "/vault/userconfig/vault/vault.crt"
          tls_key_file  = "/vault/userconfig/vault/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault/vault.ca"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 3
            server_stabilization_time = "10s"
          }

        }

[DavidRBanks]

For anyone running across this who happens to be following the documentation in a cluster running 1.22 or greater, with the changes to the certificate API, you will need to make some changes.

First, for the csr.yaml file you will need a signerName. Reference the documentation here for a TLS cert. You will want to use signerName: kubernetes.io/kubelet-serving for the server cert. That being said, you'll need to modify the command in the vault documentation to include the organization and common name requirements for that signerName, as mentioned here.

You'll know if you're dealing with this problem if your certificate immediately goes to the Approved,Failed status when you approve the csr. The way I solved this was to modify the command to create the server.cert by using the following command. This sets the organization and common names to align with the requirements for this signer:

openssl req -new -key ${TMPDIR}/vault.key -subj "/O=system:nodes/CN=system:node:${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf

[minhthong582000]

For anyone running across this who happens to be following the documentation in a cluster running 1.22 or greater, with the changes to the certificate API, you will need to make some changes.

First, for the csr.yaml file you will need a signerName. Reference the documentation here for a TLS cert. You will want to use signerName: kubernetes.io/kubelet-serving for the server cert. That being said, you'll need to modify the command in the vault documentation to include the organization and common name requirements for that signerName, as mentioned here.

You'll know if you're dealing with this problem if your certificate immediately goes to the Approved,Failed status when you approve the csr. The way I solved this was to modify the command to create the server.cert by using the following command. This sets the organization and common names to align with the requirements for this signer:

openssl req -new -key ${TMPDIR}/vault.key -subj "/O=system:nodes/CN=system:node:${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf

I got this error if i combine your solution with this document.

Error checking seal status: Get "https://vault.vault.svc:8200/v1/sys/seal-status": x509: certificate signed by unknown authority

My values.yaml:

# Vault Helm Chart Value Overrides
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"

  resources:
      requests:
        memory: 512Mi
        cpu: 500m
      limits:
        memory: 512Mi
        cpu: 500m

server:
  # Use the Enterprise Image
  image:
    repository: "hashicorp/vault"
    tag: "1.9.0"

  # These Resource Limits are in line with node requirements in the
  # Vault Reference Architecture for a Small Cluster
  resources: {}

  # For HA configuration and because we need to manually init the vault,
  # we need to define custom readiness/liveness Probe settings
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60

  affinity: |

  # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
  # used to include variables required for auto-unseal.
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  # extraVolumes is a list of extra volumes to mount. These will be exposed
  # to Vault in the path `/vault/userconfig/<name>/`.
  extraVolumes:
    - type: secret
      name: vault-server-tls

  # This configures the Vault Statefulset to create a PVC for audit logs.
  # See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 5
    raft:
      enabled: true
      setNodeId: true

      config: |
        ui = true
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
        }

        storage "raft" {
          path = "/vault/data"
            retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
              leader_api_addr = "https://vault-3.vault-internal:8200"
              leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
              leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
              leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
              leader_api_addr = "https://vault-4.vault-internal:8200"
              leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
              leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
              leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 5
            server_stabilization_time = "10s"
          }

        }

        service_registration "kubernetes" {}

# Vault UI
ui:
  enabled: true
  serviceType: "NodePort"
  serviceNodePort: 30082
  externalPort: 8200

Am i missing something ?

[DavidRBanks]

Am i missing something ?

I think we'd have to see your certificate manifests to see how you signed and created the certificate. I used the cluster CA for my test deployment and had no issues, but the error you're receiving seems straight forward.

[phozzy]

With k8s 1.19 the apiVersion reached certificates.k8s.io/v1. So, now it requires specifying the signerName.
Since we need

spec:
  usages:
    - digital signature
    - key encipherment
    - server auth

the signerName should be kubernetes.io/kubelet-serving, right?

[avoidik]

it is possible to set specific server name which will be used in TLS handshake

retry_join {
  leader_tls_servername = "vault"
}

ref. https://www.vaultproject.io/docs/concepts/integrated-storage#autojoin-with-tls-servername

[SachinMaharana]

Hello All, thanks for all the comment which helped me in setting up vault in k8s with self-signed certificate.
I have a quick question, if i plan to expose the vault as ingress to be consumed by the external client, how can i accomplish that.
If i have a domain va.example.com and i point it to ingress-nginx loadbalancer, with pointing to vault service, will the self signed cert be a issue, since i didn't mention the va.example.com as alt names in the config?

[hsimon-hashicorp]

As the documentation has been updated, I'm going to go ahead and close this issue now. Please feel free to open a new issue if you continue to have problems with TLS and vault init. Thanks!

[suhas4kolte]

I have configured the Vault-HA with TLS on Openshift4.8.36 platform by referring below links.

https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls

#243.

But Getting error at below steps while unseal vault-1 pods

[root@tef-line-bastion hashicorp-vault]# VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" cluster-keys.json)
[root@tef-line-bastion hashicorp-vault]# kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
Key Value

---

Joined true
[root@tef-line-bastion hashicorp-vault]# kubectl exec -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
Error unsealing: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/unseal
Code: 400. Errors:

• Vault is not initialized
  command terminated with exit code 2
  [root@tef-line-bastion hashicorp-vault]#

Snippet vault-1 >>>

2022-11-18T11:45:10.139Z [INFO] core: security barrier not initialized
2022-11-18T11:45:15.138Z [INFO] core: security barrier not initialized
2022-11-18T11:45:18.070Z [INFO] core: security barrier not initialized
2022-11-18T11:45:18.070Z [INFO] core: security barrier not initialized
2022-11-18T11:45:18.081Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2022-11-18T11:45:18.087Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-0.vault-internal:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT http://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 400. Raw Message:
|
| Client sent an HTTP request to an HTTPS server.

2022-11-18T11:45:20.141Z [INFO] core: security barrier not initialized
2022-11-18T11:45:25.140Z [INFO] core: security barrier not initialized
2022-11-18T11:45:30.139Z [INFO] core: security barrier not initialized
2022-11-18T11:45:35.138Z [INFO] core: security barrier not initialized

Let me know why vault-1 not initialized after joining to vault-0.

Any suggestion on this.