Using Vault as an issuer doesn't work

[SantiMunoz]

Using Vault as an issuer doesn't seem to generate correct certificates for the pods. When I try to make a request from one service to another I get this error:

~ root@netutils-6bc77c956d-jd8dz:/# curl productpage:9080
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

~ istio-proxy@netutils-6bc77c956d-jd8dz:/$ openssl s_client -showcerts -connect productpage:9080
CONNECTED(00000005)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 313 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

The Vault issuer is correctly installed and istio has been installed pointing to the istio-csr service as described in the README of this repository.

~ kubectl get clusterissuers -n cert-manager -o wide
NAME           READY   STATUS           AGE
vault-issuer   True    Vault verified   3d21h

In the istiod logs I can see how the Vault CA is correctly loaded

info	validationController	Reconcile(enter): CABundle changed
info	x509 cert - Issuer: "CN=dev eu Services Intermediate CA,OU=XX,O=XXXX,L=XXX,ST=XXX,C=XX", Subject: "CN=istiod.istio-system.svc", SN: 4df3c56e250e8464b7466308fa431cb95819f321, NotBefore: "2021-09-27T12:10:48Z", NotAfter: "2021-09-27T13:11:18Z"
info	x509 cert - Issuer: "CN=dev eu Services Root CA,OU=XX,O=XX,L=XXX,ST=XXX,C=XX", Subject: "CN=dev eu Services Intermediate CA,OU=XX,O=XXX,L=XXX,ST=XXX,C=XX", SN: 5688b4cf53441cd2ae9b18f7d617211d4588912a, NotBefore: "2019-04-01T14:46:00Z", NotAfter: "2029-03-29T14:46:00Z"
info	Istiod certificates are reloaded

In the istio-csr logs I can see the following message. I'm not sure if the namespace defined here is right or not, istio-csr is in the cert-manager ns, while istiod is in istio-system and productpage in the default namespace.

info	klog	cert-manager "msg"="signed CertificateRequest" "identity"="spiffe://cluster.local/ns/default/sa/bookinfo-productpage" "name"="istio-csr-6bqpb" "namespace"="istio-system" 

Another interesting fact is that when I sniff the traffic in the istio-proxy sidecars of the origin and destination pods I can see the request's body and response encrypted, but the client still reporting the CERTIFICATE_VERIFY_FAILED issue.

I'm using cert-manager 1.5.3, istio-csr 0.3.0 and Kubernetes 1.20.5

[JoshVanL]

Hi @SantiMunoz, can you please confirm that the signed certificates have both "client auth", and "server auth" key usages set. This is configured through vault.

[SantiMunoz]

Hello @JoshVanL, thanks for the quick response!
I've issued a certificate directly from vault with the kubernetes role I use to sign them in Istio, and I can see that the new certificate has the client and server extended key usages

~ vault write dynamic/certificates/internal_ca_dev/issue/kubernetes common_name=www.example.com ttl=72h

.......
X509v3 Key Usage: critical
     Digital Signature, Key Encipherment, Key Agreement
X509v3 Extended Key Usage:
     TLS Web Server Authentication, TLS Web Client Authentication
........

I can't find the way to check this for the CSRs from istio as I don't find them in the kubernetes cluster.

[JoshVanL]

If you enable the --preserve-certificate-requests option on istio-csr, it will preserve the CertificateRequests it creates which will include the signed certificate.

It could also potentially be a CA problem. Is the CA that is distributed within the configmaps in every namespace what you are expecting?

[SantiMunoz]

This is an example of the CSR for the productpage service

~ kubectl describe certificaterequests istio-csr-wctgb -n istio-system
Name:         istio-csr-wctgb
Namespace:    istio-system
Labels:       <none>
Annotations:  istio.cert-manager.io/identities: spiffe://cluster.local/ns/default/sa/bookinfo-productpage
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2021-09-27T17:20:44Z
  Generate Name:       istio-csr-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:istio.cert-manager.io/identities:
        f:generateName:
      f:spec:
        .:
        f:duration:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
        f:usages:
    Manager:      cert-manager-istio-csr
    Operation:    Update
    Time:         2021-09-27T17:20:44Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:ca:
        f:certificate:
        f:conditions:
    Manager:         controller
    Operation:       Update
    Time:            2021-09-27T17:20:44Z
  Resource Version:  3201968
  UID:               072cc89e-b530-4922-bbd5-33dee71b1738
Spec:
  Duration:  1h0m0s
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   vault-issuer
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3FqQ0NBWklDQVFBd0N6RUpNQWNHQTFVRUNoTUFNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QQpNSUlCQ2dLQ0FRRUE5N0lDM3c3bGNWUUxKaHFROTNEZDJjRUdBZWgwZ3pMVDJlZjBOaytaeCtsQzlyRlVab3JmCkNtQzhVc0llakd3NENJQnV3U2l1L1pGMHBrc3dMY0FJdHJvRjdDZC9YQnNOTWd1R0VyencrSUtGblNYYkFiUVQKN0VrWXdpdDJPQkRzSXRqc3ZOdDB0VDJ4Z1FLR1hSa3JOWTFXS09sUEVOeExHdk8zN1E4QU5nTm9VMmRFMjlZVgp6b1N5NTcwb1FXTFU0NmgxWG56MStDWG4ybjJ5SlkzT3ZUSEI1OVgxSm1oRlIrWlM1a0x5bi95Ym95OTV6UGVKCnRwaWtlQkZ6MnBHYnVEcTMxT0V0am92bFQ2ZkhpSjB2STdPRWlmODVMc2NYTHhrRE4vNDBxSWhTTHU2aFJEck4KQndTN1padWNYSVJzV1MvMG5jbWZEZlQ3QkJacDZqK244UUlEQVFBQm9Gb3dXQVlKS29aSWh2Y05BUWtPTVVzdwpTVEJIQmdOVkhSRUJBZjhFUFRBN2hqbHpjR2xtWm1VNkx5OWpiSFZ6ZEdWeUxteHZZMkZzTDI1ekwyUmxabUYxCmJIUXZjMkV2WW05dmEybHVabTh0Y0hKdlpIVmpkSEJoWjJVd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFDbkcKRFQ5cC9oZ25lL3hmZ2s1NU5zTEwzUnNIL1RjWTVXT2RNUDVtZGFmMVNkN29kTnkxM01QU3ZvV0tNSnMxdXhzbQpUTEtkZytMVmNLRVV1bU1mNVJ2VEpFVm9TTFEraE8xeTRFVzFzbEJMRlVpT0pjdWlKTGFYMElxTXkwRUd5ZmxCClJhTjR1bGp1akxpbGNpWmZYRkhVVElkbDJzTW5qN2ZFRnZFWkhmbno5RG1OR1FNY2k0OUVreUZHb1BzYkVZTmQKUUlmSnpkZEk4bnUxRk9CUWpzQ2VCQUVpM3JCcGJvL1pOcTdubFY0Qkh5ejIyd29GdHdtS1FLU1FJeVZxN2E2egpiNnByTkZWOUo5N1oyT3UrUVB0VUdiWng3am5XRVhtWDY1WkxzNnFZaTJQQUJDVVRranpVTHpuaWw2MTZsbkpkClh5L09MaDc1Z2tjbm5CaUszK0U9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  UID:      2824b52e-346b-4275-87ac-62e789d5153f
  Usages:
    client auth
    server auth
  Username:  system:serviceaccount:cert-manager:cert-manager-istio-csr
Status:
  Ca:       XXX
  Certificate:  XXX
  Conditions:
    Last Transition Time:  2021-09-27T17:20:44Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2021-09-27T17:20:44Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason             Age   From          Message
  ----    ------             ----  ----          -------
  Normal  cert-manager.io    40s   cert-manager  Certificate request has been approved by cert-manager.io
  Normal  CertificateIssued  40s   cert-manager  Certificate fetched from issuer successfully

The certificate signed by Vault we can see the following data, and signed by the correct intermediate:

X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                D9:A7:B8:EE:3A:98:DF:74:AB:11:77:90:D8:87:DD:AB:EA:34:DB:92
            X509v3 Authority Key Identifier:
                keyid:59:C2:CC:FE:B5:A3:E0:D0:49:D9:85:2E:E6:FC:A0:D3:81:18:D7:0E

            Authority Information Access:
                CA Issuers - URI:https://127.0.0.1:8200/v1/dynamic/certificates/internal_ca_dev/ca

            X509v3 Subject Alternative Name: critical
                URI:spiffe://cluster.local/ns/default/sa/bookinfo-productpage
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:https://127.0.0.1:8200/v1/dynamic/certificates/internal_ca_dev/crl

In all the namespaces I can see the istio-ca-root-cert configmap with the same CA we find in the certificate requests.

[JoshVanL]

Hmm, could you check that you have the correct certificate chain you are expecting on the istio proxy itself? You should have the full chain, including the root in the chain, and the root in the root cert field.

$ istioctl pc s <pod-name> -o json  

[SantiMunoz]

I've managed to fix it!
After checking the full chain (thanks for the tip!) I noticed that the intermediate was duplicated and the root CA was missing.
I also noticed that the istio-ca-root-cert had the intermediate instead of the root CA, probably because in Vault is the Intermediate signing the certificates. Is this behaviour in istio-csr intended?

After redeploying istio-csr pinning the root CA I can correctly see the CA propagated to istio-ca-root-cert and the service-to-service communication works as expected.
Thanks a lot for the help on this!

[JoshVanL]

No problem @SantiMunoz, glad you got things working!

probably because in Vault is the Intermediate signing the certificates. Is this behaviour in istio-csr intended?

This is actually coming from Vault. By default, Vault will only respond with the intermediate when a request is signed through cert-manager. It is possible to make Vault return the full chain, but this method works equally well :)

[Fr0zenBoy]

@SantiMunoz I'm having the same problem, but I'm using an external issuer, can you show me the configuration to do this:
After redeploying istio-csr pinning the root CA I can correctly see the CA propagated to istio-ca-root-cert and the service-to-service communication works as expected.
Thanks a lot for the help on this!