Issues with Vault plugins intergation

[SachinMaharana]

Describe the bug
I am having issues relating to setup plugin for vault. While vault works fine and i could get the secrets as injector with tls enabled, but setting up plugins is hitting a roadblock.
While writing the config file for plugin, i get the error.

To Reproduce

vault write sys/plugins/catalog/secret/op-connect \
sha_256="$(sha256sum /vault/data/plugins/op-connect | cut -d " " -f1)" \
command="op-connect -ca-cert=/vault/userconfig/vault-server-tls/vault.ca"


vault secrets enable --plugin-name='op-connect' --path="op" plugin

// I get a issue here
vault write op/config @op-connect-config.json

secrets.op-connect.op-connect_96388717.op-connect.op-connect: plugin tls init: error="error during token unwrap request: Put \"https://10.4.1.8:8200/v1/sys/wrapping/unwrap\": x509: certificate is valid for 127.0.0.1, 34.xx.xx.xx, not 10.4.1.8" timestamp=2022-02-17T18:06:44.542Z
2022-02-17T18:06:44.547Z [INFO]  http: TLS handshake error from 10.4.1.8:43720: remote error: tls: bad certificate

Not sure why the API_ADDR is the pod ip, which is not present in the SAN of the csr as seen below

My CSR config is

cat <<EOF >${TMPDIR}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_INTERNAL_SVC}
DNS.2 = *.${NAMESPACE}.svc.cluster.local
DNS.3 = *.${VAULT_INTERNAL_SVC}.${NAMESPACE}.svc.cluster.local
DNS.4 = vault.${NAMESPACE}.svc.cluster.local
DNS.5 = vault-agent-injector-svc
DNS.6 = vault-agent-injector-svc.vault
DNS.7 = vault-agent-injector-svc.vault.svc
DNS.8 = vault-agent-injector-svc.vault.svc.cluster.local
DNS.9 = vault
DNS.10 = vault.${NAMESPACE}
DNS.11 = vault.${NAMESPACE}.svc
IP.1 = 127.0.0.1
IP.2 = 34.xx.xx.xx
EOF

Any hint would be of great help. Thanks

Steps to reproduce the behavior:

1. Run vault write ...
2. Run vault login....
3. See error

Expected behavior
Expected the plugin to work

Environment:

• Vault Server Version (retrieve with vault status):

vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.9.3
Storage Type            raft
Cluster Name            vault-cluster-9145a1af
Cluster ID              185abbbc-2241-3420-93b4-8e094ee35709
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-02-17T17:42:03.729233783Z
Raft Committed Index    246
Raft Applied Index      246

• Vault CLI Version (retrieve with vault version):

Vault v1.9.3 (7dbdd57243a0d8d9d9e07cd01eb657369f8e1b8a)

• Server Operating System/Architecture:
  Kubernetes with vault helm chart, HA, Integrated Storage, TLS enabled
  Vault server configuration file(s):

global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  agentImage:
    repository: "hashicorp/vault"
    tag: "1.9.3"

csi:
  enabled: true
  volumes:
    - name: vault-server-tls
      secret:
        secretName: vault-server-tls
  volumeMounts:
    - name: vault-server-tls
      mountPath: /mnt/tls
      readOnly: true

server:
  extraInitContainers:
    - name: op-connect
      image: "alpine"
      command: [sh, -c]
      args:
        - >
          cd /tmp;
          wget -O op-connect.zip
          https://github.com/1Password/vault-plugin-secrets-onepassword/releases/download/v1.0.0/vault-plugin-secrets-onepassword_1.0.0_linux_amd64.zip;
          unzip op-connect.zip;
          mv vault-plugin-secrets-onepassword_v1.0.0 /vault/data/plugins/op-connect;
          chmod +x /vault/data/plugins/op-connect;
          touch op-connect-config.json;
          mv op-connect-config.json /vault/data/plugins;
      volumeMounts:
        - name: plugins
          mountPath: /vault/data/plugins
  volumes:
    - name: plugins
      emptyDir: {}
  volumeMounts:
    - mountPath: /vault/data/plugins
      name: plugins
  image:
    repository: "hashicorp/vault"
    tag: "1.9.3"

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault-server-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-server-tls/vault.key
    # GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/vault-gcs/vault_gcs_key.json

  extraVolumes:
    - type: secret
      name: vault-server-tls
    # - type: secret
    #   name: vault-gcs
    #   path: "/vault/userconfig"

  auditStorage:
    enabled: true
    mountPath: "/vault/audit"
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false

      config: |
        ui = true
        api_addr = "http://POD_IP:8200"
        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"

          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
        }
        service_registration "kubernetes" {}
        disable_mlock = true
        plugin_directory = "/vault/data/plugins"


        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 3
            server_stabilization_time = "10s"
          }

        }
ui:
  enabled: true
  serviceType: "NodePort"
  serviceNodePort: 30082
  externalPort: 8200

Additional context
Add any other context about the problem here.

[ccapurso]

Hi, @SachinMaharana. Thank you for your engagement! Our use of GitHub issues is meant to enable the Vault community the to submit feature requests and bug reports. The Vault Discuss forum is the best venue to seek feedback on general usage issues. It appears that you have submitted a similar issue there already. As such, I'm going to close this issue.

[phishbandana]

Can we open this back up or submit this to support? The question here isn't answered: similar issue

[maxb]

The problem reported here relates in part to the issue #15070 that I opened.

Specifically, Vault plugins connect back to the api_addr of the Vault that started them, as part of their initialization.

This is an obscure detail that I haven't found documented anywhere, and needed to learn from the source code.

Therefore what is going wrong here, is that the user is attempting to run a Vault configured with an SSL certificate that is not valid for the hostname/IP address in its configured api_addr.

My issue #15070 goes into further detail about why I think plugins should not use api_addr as part of their initialization.