CA isn't properly propagated when using Vault

[SantiMunoz]

Hello again, this is a related issue I have after setting up Vault as a CA #105. In the previous issue I was able to create a valid chain by specifying the CA in istio-csr using app.tls.rootCAFile.

Now I'm trying to setup the cluster from scratch and I've found an issue on the certificate chain used by istiod or istio-csr, not sure yet.
The problem I'm facing is that the istio ingress and egress gateway's pods don't get healthy and in the logs we can see an error saying that the certificate is signed with an untrusted authority. At this point I can't tell to which service is trying to reach, I would assume it's istiod or istio-csr.

I've validated that the istio-ca-root-cert is correctly propagated to all the namespaces with the same CA I loaded in istio-csr using app.tls.rootCAFile.
What I've also noticed is that at this point we have two CertificateRequests generated, istio-csr-g84kt and istiod-ww4vx, both of them have the Intermediate in the Ca field instead of the CA I loaded in istio-csr.

My assumption here is that istio-ingressgateway correctly trust the CA loaded in istio-ca-root-cert, but the TLS handshake with istiod or istio-csr is failing because they are returning the Intermediated as the CA.
Additionally, to validate the previous assumption, any deployment fails to create pods with a handshake error as well.

I've also tried to load the CA chain in istio-csr, by loading a file containing the Intermediate and the CA using app.tls.rootCAFile. With this the ingress and egress gateway pods were able to successfully run and normal deployments were able to schedule their pods, but the certificate chain in the istio-proxy of the application now is invalid, as it has the intermediate duplicated.

I think the problem here is that the CA loaded from istio-csr is not correctly propagated to these firsts CertificateRequests.

istio-csr config used:

kubectl create secret generic istio-root-ca --from-file=ca.pem=ca.pem -n cert-manager

helm install -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr \
--set app.certmanager.issuer.name=vault-issuer \
--set app.certmanager.issuer.kind=ClusterIssuer \
--set app.certmanager.namespace=istio-system \
--set app.certmanager.preserveCertificateRequests=true \
--set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \
--set "volumeMounts[0].name=root-ca" \
--set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
--set "volumes[0].name=root-ca" \
--set "volumes[0].secret.secretName=istio-root-ca" \
--version 0.3.0

Istio Operator config used

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  profile: "demo"
  hub: gcr.io/istio-release
  values:
    global:
      # Change certificate provider to cert-manager istio agent for istio agent
      caAddress: cert-manager-istio-csr.cert-manager.svc:443
      proxy:
        privileged: true # allows to tcpdump envoy proxy connections
  components:
    pilot:
      k8s:
        env:
          # Disable istiod CA Sever functionality
        - name: ENABLE_CA_SERVER
          value: "false"
        overlays:
        - apiVersion: apps/v1
          kind: Deployment
          name: istiod
          patches:

            # Mount istiod serving and webhook certificate from Secret mount
          - path: spec.template.spec.containers.[name:discovery].args[-1]
            value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
          - path: spec.template.spec.containers.[name:discovery].args[-1]
            value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
          - path: spec.template.spec.containers.[name:discovery].args[-1]
            value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"

          - path: spec.template.spec.containers.[name:discovery].volumeMounts[-1]
            value:
              name: cert-manager
              mountPath: "/etc/cert-manager/tls"
              readOnly: true
          - path: spec.template.spec.containers.[name:discovery].volumeMounts[-1]
            value:
              name: ca-root-cert
              mountPath: "/etc/cert-manager/ca"
              readOnly: true

          - path: spec.template.spec.volumes[-1]
            value:
              name: cert-manager
              secret:
                secretName: istiod-tls
          - path: spec.template.spec.volumes[-1]
            value:
              name: ca-root-cert
              configMap:
                defaultMode: 420
                name: istio-ca-root-cert

[JoshVanL]

Hi @SantiMunoz, can you verify that the CA being propagated is in-fact the CA that you are intending to propagate, and that the resulting certificates contain a full valid chain.

$ openssl verify -verbose -CAfile <(cat Intermediate.pem RootCert.pem) IstioCert.pem

RootCert.pem should contain the CA that is being propergated.
Intermediate.pem should be the intermediates as they appear in the CertificateRequest, not including the leaf certificate
IstioCert.pem the leaf certificate

---

I think the problem here is that the CA loaded from istio-csr is not correctly propagated to these firsts CertificateRequests.

The CertificateRequests are signed and updated by the configured cert-manager Issuer. istio-csr is only responsible and capable of creating the resource and reading back what it is updated with.

[SantiMunoz]

@JoshVanL thanks again for your tips, I realised I was loading the wrong CA in istio-csr. Everything works as expected with the correct CA :)