Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLS handshake error #289

Open
RakeshRaj97 opened this issue Aug 30, 2021 · 9 comments
Open

TLS handshake error #289

RakeshRaj97 opened this issue Aug 30, 2021 · 9 comments

Comments

@RakeshRaj97
Copy link

RakeshRaj97 commented Aug 30, 2021 •
edited
Loading

I installed vault in HA mode using self-signed certificate. Process I followed,

Step 1: Create key & certificate using Kubernetes CA

Define environment variables

SERVICE=vault-server-tls

NAMESPACE=vault

SECRET_NAME=vault-server-tls

TMPDIR=/tmp

Create a key for Kubernetes to sign

openssl genrsa -out ${TMPDIR}/vault.key 2048

Create a Certificate Signing Request (CSR)

cat <<EOF >${TMPDIR}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE}
DNS.2 = ${SERVICE}.${NAMESPACE}
DNS.3 = ${SERVICE}.${NAMESPACE}.svc
DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
DNS.5 = vault-0.vault-internal
DNS.6 = vault-1.vault-internal
DNS.7 = vault-2.vault-internal
DNS.8 = vault-0.vault-internal.svc
DNS.9 = vault-1.vault-internal.svc
DNS.10 = vault-2.vault-internal.svc
DNS.11 = vault-0.vault-internal.svc.cluster.local
DNS.12 = vault-1.vault-internal.svc.cluster.local
DNS.13 = vault-2.vault-internal.svc.cluster.local
DNS.14 = vault-0
DNS.15 = vault-1
DNS.16 = vault-2
DNS.17 = vault-agent-injector-svc
DNS.18 = vault-agent-injector-svc.${NAMESPACE}
DNS.19 = vault-agent-injector-svc.${NAMESPACE}.svc
DNS.20 = vault-agent-injector-svc.${NAMESPACE}.svc.cluster.local
IP.1 = 127.0.0.1
EOF

Create a CSR

openssl req -new -key ${TMPDIR}/vault.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf

Create the certificate

export CSR_NAME=vault-csr
cat <<EOF >${TMPDIR}/csr.yaml
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${CSR_NAME}
spec:
  groups:
  - system:authenticated
  request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

Send the CSR to Kubernetes

kubectl create -f ${TMPDIR}/csr.yaml

Approve the CSR in Kubernetes

kubectl certificate approve ${CSR_NAME}

Step 2: Sort key, cert, and Kubernetes CA into Kubernetes secrets store

Retrieve the certificate

serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')

Write the certificate out to a file

echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt

Create the namespace

kubectl create namespace ${NAMESPACE}

Store the key, cert, and Kubernetes CA into Kubernetes secrets

kubectl create secret generic ${SECRET_NAME} \
        --namespace ${NAMESPACE} \
        --from-file=vault.key=${TMPDIR}/vault.key \
        --from-file=vault.crt=${TMPDIR}/vault.crt \
        --from-file=vault.ca=${TMPDIR}/vault.ca

Step 3: Install using Helm

override-values.yaml

global:
  enabled: true
  tlsDisable: false

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault-server-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-server-tls/vault.key

  extraVolumes:
  - type: secret
    name: vault-server-tls

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false
      config: |
        ui = true
        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = ""0.0.0.0:8201"
          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
         # tls_require_and_verify_client_cert = 1
        }

        storage "raft" {
          path = "/vault/data"
            retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 3
            server_stabilization_time = "10s"
          }
        }
        service_registration "kubernetes" {}

Step 4: Initialised and unsealed vault

kubectl get pods --selector='app.kubernetes.io/name=vault'
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          1m49s
vault-1                                 1/1     Running   0          1m49s
vault-2                                 1/1     Running   0          1m49s
vault-agent-injector-58446c77f4-7n6jp   1/1     Running   0          1m49s

Step 5: Create dummy secrets and configure K8s authentication

vault secrets enable -path=secret kv-v2

vault kv put secret/devwebapp/config username='giraffe' password='salsa'

vault auth enable kubernetes

vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=devwebapp \
        ttl=24h

Step 6: Deploy web application

kubectl create sa internal-app -n default

apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  namespace: default
  labels:
    app: devwebapp
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-status: "update"
   #vault.hashicorp.com/tls-skip-verify: "true" 
    vault.hashicorp.com/role: "devweb-app-1"
    vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"

    vault.hashicorp.com/tls-secret: "vault-server-tls"
    vault.hashicorp.com/ca-cert: "/vault/tls/vault.ca"
    vault.hashicorp.com/client-cert: "/vault/tls/vault.crt"
    vault.hashicorp.com/client-key: "/vault/tls/vault.key"
spec:
  serviceAccountName: internal-app
  containers:
    - name: devwebapp
      image: jweissig/app:0.0.1

Logs:

k logs devwebapp -c vault-agent-init

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.8.1
             Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d

2021-08-30T01:38:17.530Z [INFO]  sink.file: creating file sink
2021-08-30T01:38:17.530Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-08-30T01:38:17.531Z [INFO]  sink.server: starting sink server
2021-08-30T01:38:17.531Z [INFO]  template.server: starting template server
2021-08-30T01:38:17.531Z [INFO]  auth.handler: starting auth handler
2021-08-30T01:38:17.531Z [INFO]  auth.handler: authenticating
2021-08-30T01:38:17.532Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-30T01:38:17.534Z [INFO] (runner) creating watcher
2021-08-30T01:39:17.532Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2021-08-30T01:39:18.533Z [INFO]  auth.handler: authenticating
2021-08-30T01:40:18.534Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.64s
2021-08-30T01:40:20.178Z [INFO]  auth.handler: authenticating
2021-08-30T01:41:20.181Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=3.09s
2021-08-30T01:41:23.277Z [INFO]  auth.handler: authenticating
2021-08-30T01:42:23.277Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4.66s
2021-08-30T01:42:27.944Z [INFO]  auth.handler: authenticating
2021-08-30T01:43:27.945Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=8.98s
2021-08-30T01:43:36.932Z [INFO]  auth.handler: authenticating
2021-08-30T01:44:36.933Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=17.12s
2021-08-30T01:44:54.064Z [INFO]  auth.handler: authenticating
2021-08-30T01:45:54.066Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=30.22s
2021-08-30T01:46:24.290Z [INFO]  auth.handler: authenticating
2021-08-30T01:47:24.294Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=59.39s
2021-08-30T01:48:23.686Z [INFO]  auth.handler: authenticating
2021-08-30T01:49:23.688Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1m51.4s

k logs vault-0

2021-08-27T06:21:10.953Z [INFO]  http: TLS handshake error from 10.244.15.1:59366: remote error: tls: bad certificate
2021-08-27T06:21:12.192Z [INFO]  http: TLS handshake error from 10.244.15.1:59372: remote error: tls: bad certificate
2021-08-27T06:21:18.538Z [INFO]  http: TLS handshake error from 10.244.15.1:59390: remote error: tls: bad certificate
2021-08-27T06:21:45.250Z [INFO]  http: TLS handshake error from 10.244.15.1:59454: remote error: tls: bad certificate
2021-08-27T06:22:12.932Z [INFO]  http: TLS handshake error from 10.244.15.1:47366: remote error: tls: bad certificate
2021-08-27T06:22:24.547Z [INFO]  http: TLS handshake error from 10.244.15.1:59580: remote error: tls: bad certificate
2021-08-27T06:22:29.662Z [INFO]  http: TLS handshake error from 10.244.15.1:59592: remote error: tls: bad certificate
2021-08-27T06:22:32.423Z [INFO]  http: TLS handshake error from 10.244.15.1:47502: remote error: tls: bad certificate
2021-08-27T06:23:08.416Z [INFO]  http: TLS handshake error from 10.244.15.1:60062: remote error: tls: bad certificate
2021-08-27T06:23:21.253Z [INFO]  http: TLS handshake error from 10.244.15.1:60096: remote error: tls: bad certificate
2021-08-27T06:23:54.774Z [INFO]  http: TLS handshake error from 10.244.15.1:60176: remote error: tls: bad certificate
2021-08-27T06:24:10.829Z [INFO]  http: TLS handshake error from 10.244.15.1:60244: remote error: tls: bad certificate
2021-08-27T06:24:21.824Z [INFO]  http: TLS handshake error from 10.244.15.1:60272: remote error: tls: bad certificate
2021-08-27T06:24:44.709Z [INFO]  http: TLS handshake error from 10.244.15.1:60338: remote error: tls: bad certificate
2021-08-27T06:24:55.447Z [INFO]  http: TLS handshake error from 10.244.15.1:60366: remote error: tls: bad certificate
2021-08-27T06:25:28.973Z [INFO]  http: TLS handshake error from 10.244.15.1:60510: remote error: tls: bad certificate
2021-08-27T06:26:06.380Z [INFO]  http: TLS handshake error from 10.244.15.1:60626: remote error: tls: bad certificate
2021-08-27T06:26:17.813Z [INFO]  http: TLS handshake error from 10.244.15.1:60662: remote error: tls: bad certificate
2021-08-27T06:26:50.514Z [INFO]  http: TLS handshake error from 10.244.15.1:60738: remote error: tls: bad certificate
2021-08-27T06:27:21.144Z [INFO]  http: TLS handshake error from 10.244.15.1:60846: remote error: tls: bad certificate
2021-08-27T06:27:23.455Z [INFO]  http: TLS handshake error from 10.244.15.1:60856: remote error: tls: bad certificate
2021-08-27T06:27:31.615Z [INFO]  http: TLS handshake error from 10.244.15.1:49694: remote error: tls: bad certificate
2021-08-27T06:28:07.956Z [INFO]  http: TLS handshake error from 10.244.15.1:49992: remote error: tls: bad certificate
2021-08-27T06:28:17.455Z [INFO]  http: TLS handshake error from 10.244.15.1:50054: remote error: tls: bad certificate
2021-08-27T06:28:28.550Z [INFO]  http: TLS handshake error from 10.244.15.1:32806: remote error: tls: bad certificate
2021-08-27T06:28:29.763Z [INFO]  http: TLS handshake error from 10.244.15.1:32810: remote error: tls: bad certificate
2021-08-27T06:28:30.766Z [INFO]  http: TLS handshake error from 10.244.15.1:50140: remote error: tls: bad certificate
2021-08-27T06:29:26.511Z [INFO]  http: TLS handshake error from 10.244.15.1:32978: remote error: tls: bad certificate
2021-08-27T06:29:43.549Z [INFO]  http: TLS handshake error from 10.244.15.1:33022: remote error: tls: bad certificate
2021-08-27T06:29:47.176Z [INFO]  http: TLS handshake error from 10.244.15.1:33028: remote error: tls: bad certificate
2021-08-27T06:31:40.219Z [INFO]  http: TLS handshake error from 10.244.15.1:33392: remote error: tls: bad certificate
2021-08-27T06:31:47.798Z [INFO]  http: TLS handshake error from 10.244.15.1:33418: remote error: tls: bad certificate
2021-08-27T06:33:05.083Z [INFO]  http: TLS handshake error from 10.244.15.1:33654: remote error: tls: bad certificate
2021-08-27T06:33:07.151Z [INFO]  http: TLS handshake error from 10.244.15.1:33658: remote error: tls: bad certificate
2021-08-27T06:33:11.361Z [INFO]  http: TLS handshake error from 10.244.15.1:33670: remote error: tls: bad certificate
2021-08-27T06:33:28.173Z [INFO]  http: TLS handshake error from 10.244.15.1:52314: remote error: tls: bad certificate
2021-08-27T06:33:52.094Z [INFO]  http: TLS handshake error from 10.244.15.1:33766: remote error: tls: bad certificate

Resources created

pod/devwebapp                               0/2     Init:0/1   0          6m45s
pod/vault-0                                 1/1     Running    0          2d20h
pod/vault-1                                 1/1     Running    0          2d20h
pod/vault-2                                 1/1     Running    0          2d20h
pod/vault-agent-injector-58446c77f4-7n6jp   1/1     Running    0          2d20h

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   10.111.201.114   <none>        8200/TCP,8201/TCP   2d20h
service/vault-active               ClusterIP   10.100.17.194    <none>        8200/TCP,8201/TCP   2d20h
service/vault-agent-injector-svc   ClusterIP   10.103.55.211    <none>        443/TCP             2d20h
service/vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   2d20h
service/vault-standby              ClusterIP   10.105.249.28    <none>        8200/TCP,8201/TCP   2d20h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           2d20h

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-58446c77f4   1         1         1       2d20h

NAME                     READY   AGE
statefulset.apps/vault   3/3     2d20h

Is there anything I'm missing?
@ErgoPr0xy
Copy link

having the same issue on vault 1.8.1, gke 1.20
if you uncomment
#vault.hashicorp.com/tls-skip-verify: "true"
does it work?
I tried adding an issuer: hashicorp/vault-helm#562 and it still broke.
As a workaround I had to disable JWT issuer validation and enabled tls-skip-verify and then it came up and the pod got injected.
Using pretty much the same config as you have here and it does not work, get the exact same errors.
I'll do some more troubleshooting.

@avoidik
Copy link

avoidik commented Jan 20, 2022

it is possible to set specific server name which will be used in TLS handshake

retry_join {
  leader_tls_servername = "vault"
}

ref. https://www.vaultproject.io/docs/concepts/integrated-storage#autojoin-with-tls-servername

@panki989
Copy link

@avoidik This works. Thanks !!!

@ethan256
Copy link

ethan256 commented Jun 8, 2022

add annotation as follow:
vault-hashicorp-com-tls-server-name: "vault-server-tls"

@tirelibirefe
Copy link

vault-hashicorp-com-tls-server-name: "vault-server-tls"

to where?

@saintmalik
Copy link

has anyone been able to configure the tls? mind sharing how?

@ethan256
Copy link

ethan256 commented Mar 6, 2023 •
edited
Loading

has anyone been able to configure the tls? mind sharing how?

I can use the following configuration and it works fine

global:
  enabled: true
  tlsDisable: false

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  # extraVolumes is a list of extra volumes to mount. These will be exposed
  # to Vault in the path `/vault/userconfig/<name>/`.
  extraVolumes:
  - type: secret
    name: vault-server-tls

  standalone:
    enabled: false

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        log_format = "json"

        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"

          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
        }

        storage "raft" {
          path = "/vault/data"
            retry_join {
            leader_tls_servername = "vault-server-tls"
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          retry_join {
            leader_tls_servername = "vault-server-tls"
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          retry_join {
            leader_tls_servername = "vault-server-tls"
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 3
            server_stabilization_time = "10s"
          }
        }
        service_registration "kubernetes" {}

@saintmalik
Copy link

Thanks, I actually meant how I can also create the TLS certs that would be mounted instead, I guess self signed tla cert

@saintmalik
Copy link

saintmalik commented Mar 6, 2023 •
edited
Loading

Thanks, i just followed this step now, but there seems to be an issue,the instruction says confirm

 kubectl get csr vault.svc
NAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITION
vault.svc   16s   kubernetes.io/kubelet-serving   minikube-user   100d         Approved,Issued

mine is only showing approved and no Issued, hence running returns empty vault.crt file

Screenshot 2023-03-07 at 12 05 02 AM

kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ErgoPr0xy @avoidik @panki989 @saintmalik @tirelibirefe @ethan256 @RakeshRaj97