Cannot specify root CA for external vault over HTTPS

[jobec]

I'm trying to setup a vault injector setup on an OpenShift cluster, but when my example pod starts, the vault agent in the init container complains it doesn't trust the CA.

2020-06-25T10:18:05.843Z [ERROR] auth.handler: error authenticating: error="Put https://vault:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.171067192

I followed all sorts of guides:

• https://learn.hashicorp.com/vault/kubernetes/external-vault
• https://www.vaultproject.io/docs/platform/k8s/injector/installation
  I looked into the configurable values and dived into the templates and the code in https://github.com/hashicorp/vault-k8s but I cannot seem to figure out how to have the injector set the VAULT_CACERT on the init/sidecar container.

The examples also show setting it up over HTTP (😱) and don't mention HTTPS anywhere when talking about an external vault.

Am I overlooking something? Or is HTTPS not supported for an external vault for an injector on kubernetes?

Btw, I do manage to make things work when doing it the old fashion way with a manual init container in my deployment config, along with a volume mount of a config map containing the root CA certificate. But I expected the injector to take care of that repetitive work.

[jobec]

Looks like it's related to hashicorp/vault-k8s#19

[jobec]

Meanwhile I found a way to get it working by creating a secret and setting the vault.hashicorp.com/ca-cert and vault.hashicorp.com/tls-secret annotations.
But this is something that needs to be done for each and every application in kubernetes while actually this should be a setting on the level of the injector helm chart. Because now it requires knowledge of the root CA in each and every pod.

Helm template values overrides

global:
  openshift: true
injector:
  enabled: true
  externalVaultAddr: "https://hashi-vault.domain.int:8200"

Demo kubernetes manifest

---
kind: Secret
apiVersion: v1
metadata:
  name: vault-tls-secret
stringData:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
type: Opaque
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: app-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-example
  template:
    metadata:
      labels:
        app: app-example
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-db-creds: testpath/testsecret
        vault.hashicorp.com/agent-inject-template-db-creds: >
          {{- with secret "testpath/testsecret" -}}
          postgres://{{ .Data.user }}:{{ .Data.password}}@postgres:5432/appdb
          {{- end }}
        vault.hashicorp.com/role: vault-injector-test-role
        vault.hashicorp.com/ca-cert: /vault/tls/ca-bundle.crt
        vault.hashicorp.com/tls-secret: vault-tls-secret
    spec:
      serviceAccountName: some-non-default-sa
      containers:
        - name: app
          image: 'bash:latest'
          command:
            - /usr/local/bin/bash
            - '-ec'
            - 'while :; do cat /vault/secrets/db-creds; sleep 5 ; done'

[jasonodonnell]

Hi @jobec,

As you discovered, mounting the CA via tis-secret is the way to configure this. Vault K8s currently doesn't support setting CA on all deployments since it would require that secret to be available in all namespaces. This has bubbled up in priority's though, so we hope to have a solution soon!

[jobec]

Isn't all this info something to put into the documentation explicitly? I can imagine quite some people struggling with this and there are quite some related issues about this.

Another option is create a custom vault agent image and inject a custom root CA into the image's bundle.

[loeken]

@jobec thank you very much for filing this issue, as i have been in the same boat as you were, trying out various different documentations - this should definately be improved on hashicorps side:

the linked issue #19's comment was the last piece of the puzzle for me:

with the injector I see that there is a vault.hashicorp.com/tls-secret annotation that will mount a Secret containing TLS certs (and that mount point can be referenced by vault.hashicorp.com/ca-cert)

[nilskch]

@jobec I had the exact same problem and I came up with a little hacky solution: In the values.yaml override the injector.agentImage with a custom Dockerimage:

injector:
  agentImage:
    repository: "<private_registry>/vault-agent-sidecar"
    tag: "<tag>"

Therefore you need to create a custom dockerimage and install the extra certs of your private CA:

FROM hashicorp/vault:1.8.0

RUN apk update && apk add bash
# Add trusted certificates
ADD ca-certificates/certs.crt /usr/local/share/ca-certificates/certs.crt
RUN chmod 644 /usr/local/share/ca-certificates/certs.crt && update-ca-certificates