No direct method to mount the Vault CA inside init/sidecar

[mbrancato]

Describe the bug
I've created an HA Vault with a cert signed by the Kube Root CA with the Helm chart. When I annotate a deployment, I get the following error from the init container:

2021-05-11T20:51:40.136Z [INFO]  auth.handler: authenticating
2021-05-11T20:51:40.195Z [ERROR] auth.handler: error authenticating: error="Put "https://vault.vault.svc:8200/v1/auth/google/login": x509: certificate signed by unknown authority" backoff=16.95s

There are some related issues about external Vaults, but this also affect the sidecar for an in-cluster Vault.

To Reproduce
Steps to reproduce the behavior:

1. Install chart
2. Annotate deployment / pod
3. See error in init container

Related to #243 #339

I think this would be addressed in either of these examples:

• https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls
• https://www.vaultproject.io/docs/platform/k8s/helm/examples/injector-tls

Expected behavior
The injector would provide a way to mount the CA cert of the Vault server CA as a trusted CA in the init container. There is no support for extraVolumes or volumes. There is at best vault.hashicorp.com/agent-copy-volume-mounts as an annotation, but that now also means you have to mount a volume in a workload container and the value of the injector is the container can be unaware of Vault.

Environment

• Kubernetes version:
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): GKE
  • Other configuration options or runtime services (istio, etc.):
• vault-helm version: 0.11.0

Chart values:

global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  agentImage:
    repository: "vault"
    tag: "1.7.1"
  agentDefaults:
    cpuLimit: "500m"
    cpuRequest: "25m"
    memLimit: "128Mi"
    memRequest: "64Mi"
  tolerations: |
    - key: vault
      operator: Exists
      effect: NoSchedule
  nodeSelector: |
    workload: vault

server:
  enabled: true
  image:
    repository: vault
    tag: "1.7.1"
  updateStrategyType: RollingUpdate
  resources:
    requests:
      memory: 64Mi
      cpu: 20m
    limits:
      memory: 768Mi
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      cert-manager.io/cluster-issuer: letsencrypt
    hosts:
      - host: vault.mydomain.org
        paths:
          - /
    tls:
      - secretName: vault-tls
        hosts:
          - vault.mydomain.org
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
    initialDelaySeconds: 60
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 120
  tolerations: |
    - key: vault
      operator: Exists
      effect: NoSchedule

  nodeSelector: |
    workload: vault

  extraEnvironmentVars:
    VAULT_CACERT: /vault/vault-server-tls/vault.ca

  volumes:
    -  name: vault-server-tls
       secret:
         secretName: vault-server-tls
  volumeMounts:
    - mountPath: /vault/vault-server-tls
      name: vault-server-tls
      readOnly: true

  service:
    enabled: true
  dataStorage:
    enabled: false
  auditStorage:
    enabled: false
  dev:
    enabled: false
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: false
    config: |
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/vault-server-tls/vault.ca"
      }

      storage "gcs" {
        bucket = "..."
        ha_enabled = true
      }

      seal "gcpckms" {
        project = "..."
        region = "us-central1"
        key_ring = "vault"
        crypto_key = "vault-key"
      }

      default_lease_ttl = "168h"
      max_lease_ttl = "720h"
      ui = true

      service_registration "kubernetes" {}

    disruptionBudget:
      enabled: true
      maxUnavailable: 1

  serviceAccount:
    create: false
    name: vault-svc
ui:
  enabled: true
  serviceType: ClusterIP
  serviceNodePort: null
  externalPort: 443

csi:
  enabled: false

Additional context
Add any other context about the problem here.

[verenion]

Any updated to this? A year later with noone else reporting the issue seems odd to me. I'm also hitting this problem, how are people working around this?

[tarekabouzeid]

Hi,
I am also having same problem, when installing this helm chart and connecting to external vault, I cannot provide the external vault CA, so in vault logs I am getting Bad certificate error when a pod tries to connect to vault and mount a secret.
Any workarounds this ?

Best Regards,

[tarekabouzeid]

Hi,
I found this and it fixed my problem, #339 (comment).
hope this is fixed in the helm chart itself instead of adding the annotations to every pod.
Thanks

[PerilousApricot]

This is a bizarre hole in the helm chart... presumably many people run vault within (secure) internal networks, w/internal CAs, so it's surprising there's no option to tell the k8s vault agent what the CA bundle is.

[lukas-pastva]

Same here, it is 2024 and still the same problem.
When running Vault on internal network with company CA, there is no easy "helm values" way to set the CA to the injector sidecars.
The only upside is, that also the application will need to have the CA(because of internal network connections eventually).
So the secret is required either way for injector and from business perspective the application/container itself.