OCP Ingress Support for Cert-Manager 1.4 #4247

charles-horel · 2021-07-23T14:32:15Z

Is your feature request related to a problem? Please describe.
OCP 1.4 ingress relies on a very specific format for the secret created by cert-manager. With recent version changes, the root ca cert has been moved from the tls.crt key to a ca.crt key, and openshift-ingress does not pick this up. This causes degradation of the operator and the full chain is not presented by ingress services.

This is an RFE as this appears to be newer intentional behavior of cert-manager:
https://cert-manager.io/docs/concepts/certificate/
"When a certificate is issued by intermediates of the CA and the Issuer knows the intermediates, the content of tls.crt will be a resulting certificate followed by a certificate chain. The certificate chain doesn’t include a root CA certificate, as it is stored in ca.crt."

Describe the solution you'd like
A parameter on the certificate object that determines how the final cert object is formatted. This could be called legacyCertificateSecretFormat bool. True would cause the entire cert chain to be included in tls.crt instead of breaking it up into ca.crt.

Describe alternatives you've considered
The only alternatives from our end would be to submit an RFE to Redhat (which we have). We are downgrading to cert-manager 1.1 in the interim.
The only other solution would be to have a custom operator update the secret or parse it into a new secret with the format we want which could be messy to maintain. This problem will persist on all openshift ingresses that use a cert tied to cert-manager 1.4 though, it would be good to sort out long term.

Additional context

Environment details (remove if not applicable):

Kubernetes version: Openshift Container Platform 4.7
Cloud-provider/provisioner:
cert-manager version: 1.4.0
Install method: e.g. helm/static manifests: helm

/kind feature

jakexks · 2021-07-23T15:28:18Z

Hi @charles-horel

We don't use openshift day to day, so I'd be happy accept an issue if it's not working. However, I tried to reproduce the openshift ingress controller not accepting TLS certs ending in an intermediate and can't reproduce it. Could you elaborate a bit more on what error you are getting?

Which issuer are you using?

Let's say I have my own root CA, and have issued an intermediate for it. Certs inside:

tls.crt:

ca.crt:

Checking my local openshift install:

❯ crc version
WARN A new version (1.29.1) has been published on https://cloud.redhat.com/openshift/create/local
CodeReady Containers version: 1.27.0+3d6bc39d
OpenShift version: 4.7.11 (not embedded in executable)

Following this documentation for configuring Ingress controllers: https://docs.openshift.com/container-platform/4.7/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress

oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
  --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'
ingresscontroller.operator.openshift.io/default patched

Trying to see if the cert works and is valid:

curl -vvv --cacert ca.crt https://console-openshift-console.apps-crc.testing/
*   Trying ::1...
* TCP_NODELAY set
* Connected to console-openshift-console.apps-crc.testing (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ca.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; L=London; O=cert-manager; CN=server.cert-manager.io
*  start date: Jul 23 15:18:00 2021 GMT
*  expire date: Jul 23 15:18:00 2022 GMT
*  subjectAltName: host "console-openshift-console.apps-crc.testing" matched cert's "console-openshift-console.apps-crc.testing"
*  issuer: C=GB; L=London; O=cert-manager; CN=I1
*  SSL certificate verify ok.

Note the ingress controller has no need for ca.crt (which is just provided for informational purposes). Clients need to already trust the root CA through another distribution method (e.g, including the cert in your docker images trust store or, mounting it from a well-known secret) or you don't really have TLS at all!

jakexks · 2021-07-23T15:30:49Z

/triage support

charles-horel · 2021-07-23T16:34:44Z

Jake thank you for getting back to us so quickly. That’s pretty interesting that you do not see the same behaviour that we do. We are also using the venafi TPP clusterissuer – it’s possible this is unique to our clusterIssuer. However as the secret format looks right I believe this is an issue with (at least our) Openshift’s interpretation of the tls secret. When we set up the venafi TPP cluster issuer and retrieve a cert it creates the tls secret exactly as we would expect. The secret looks something like this: ***@***.*** ~]$ oc get secret openshift-ingress-certificate -n openshift-ingress -o yaml apiVersion: v1 data: ca.crt: # Note - ingress will not present this root unless it is appended to tls.crt key <b64 encoded contents of root cert only> tls.crt: #Venafi TPP by default only places the end entity and intermediate entities here < <b64 encoded contents of the end entity and intermediate certs> tls.key: <b64 encoded private key> kind: Secret metadata: When we install the cert, for quite some time everything seems fine. It takes awhile for the ingress certificate to cycle, and then awhile longer after that, but the warning ‘routercertsdegraded’ begins to show in the console (hopefully the attached png makes its way through) – the root error reported by the operator is ‘x509 unrecognized certificate authority’ as it does not see the root cert. From a browser perspective, because we have trust established locally it actually doesn’t see an issue with the cert. We have to use openssl to reveal that the root entity is missing. Can you quickly run/confirm this on your test cluster? ***@***.*** ~]$ openssl s_client -showcerts -servername console-openshift-console.apps.<redact> -connect console-openshift-console.apps.<redact>:443 </dev/null CONNECTED(00000003) depth=2 CN = <redact - our prod CA> verify return:1 depth=1 DC = AD, DC = <redact>, CN = <redact> verify return:1 depth=0 C = CA, ST = <redact>, L = <redact>, O = <redact>, OU = Information Technology, CN =<redact> verify return:1 --- Certificate chain 0 s:/C=CA/ST=Alberta/L=<redact>/O=<redact>/OU=Information Technology/CN=<redact> i:/DC=AD/DC=PRD/CN=<redact - our prod CA> -----BEGIN CERTIFICATE----- END CERT ENTITY -----END CERTIFICATE----- 1 s:/DC=AD/DC=<redact>,/CN=<redact - our prod CA> i:/CN=<redact - our prod CA> -----BEGIN CERTIFICATE----- INTERMEDIATE CERT ENTITY -----END CERTIFICATE----- #note - no root cert --- Server certificate subject=/C=CA/ST=Alberta/L=<redact>/O=<redact>/OU=Information Technology/CN=<redact> issuer=/DC=AD/DC=<redact>/CN=<redact> --- Here is our lab openshift versions: ***@***.*** ~]$ oc version Client Version: 4.7.7 Server Version: 4.7.7 Kubernetes Version: v1.20.0+c8905da ***@***.*** ~]$ Charles Horel, P.Eng E: ***@***.******@***.***> From: Jake Sanders ***@***.***> Sent: Friday, July 23, 2021 9:28 AM To: jetstack/cert-manager ***@***.***> Cc: Charles Horel ***@***.***>; Mention ***@***.***> Subject: Re: [jetstack/cert-manager] OCP Ingress Support for Cert-Manager 1.4 (#4247) CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi @charles-horel<https://github.com/charles-horel> We don't use openshift day to day, so I'd be happy accept an issue if it's not working. However, I tried to reproduce the openshift ingress controller not accepting TLS certs ending in an intermediate and can't reproduce it. Could you elaborate a bit more on what error you are getting? Let's say I have my own root CA, and have issued an intermediate for it. Certs inside: tls.crt:

…

-----END CERTIFICATE-----

-----END CERTIFICATE----- ca.crt:

-----END CERTIFICATE----- Checking my local openshift install: ❯ crc version WARN A new version (1.29.1) has been published on https://cloud.redhat.com/openshift/create/local CodeReady Containers version: 1.27.0+3d6bc39d OpenShift version: 4.7.11 (not embedded in executable) Following this documentation for configuring Ingress controllers: https://docs.openshift.com/container-platform/4.7/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \

--patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}' ingresscontroller.operator.openshift.io/default patched Trying to see if the cert works and is valid: curl -vvv --cacert ca.crt https://console-openshift-console.apps-crc.testing/ * Trying ::1... * TCP_NODELAY set * Connected to console-openshift-console.apps-crc.testing (::1) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: ca.crt CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=GB; L=London; O=cert-manager; CN=server.cert-manager.io * start date: Jul 23 15:18:00 2021 GMT * expire date: Jul 23 15:18:00 2022 GMT * subjectAltName: host "console-openshift-console.apps-crc.testing" matched cert's "console-openshift-console.apps-crc.testing" * issuer: C=GB; L=London; O=cert-manager; CN=I1 * SSL certificate verify ok. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4247 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU6LYPI2H4NT2PSJL3UE3PLTZGDBZANCNFSM5A4DZ6NA>.

charles-horel · 2021-07-28T15:18:43Z

Note we opened an RFE with Redhat as well to accept/leverage the new cert format:

https://issues.redhat.com/browse/RFE-2023

absynth76 · 2021-08-27T13:02:12Z

Hi, the main issue is that ingress controller certificate is also used by openshift as a trust bundle in other operators, like the authentication one. So even if the router/ingress controller does not complain about a certificate chain not containing the Root CA, The other operators that consume the same secret are going in a degrade state and so openshift cluster is consequently degraded too. I really do not understand why you can't accept the PR feature that @SgtCoDFish created above, this could have be a potential solution to make cert-manager fully compatible with openshift.

charles-horel · 2021-08-30T22:01:53Z

I think it's possible all that is needed in order to resolve the authentication operator is to get the trust added for the root CA. I did a bit more digging and saw this, which indicates the full cert should never need be presented if the root is a trusted entity.

https://cert-manager.io/docs/concepts/certificate/
cert-manager intentionally avoids adding root certificates to tls.crt, because they are useless in a situation where TLS is being done securely. For more information, see RFC 5246 section 7.4.2 which contains the following explanation:

Because certificate validation requires that root keys be distributed independently, the self-signed certificate that specifies the root certificate authority MAY be omitted from the chain, under the assumption that the remote end must already possess it in order to validate it in any case.

I am reading this to believe that we should be adding trust to whatever store is managed by authentication. If Openshift operators aren't leveraging the cluster trust store, they should. This would mean that this issue is not an issue, and more of a configuration issue with Openshift...

To corroborate this, we did notice that our browsers (which trust the root CA) did not complain about the console certificate even though it only contained the intermediate and end entities!

absynth76 · 2021-08-31T06:02:44Z

And again I'll repeat myself, MAY goes in two directions, it means can or cannot, it means a flag should be implemented to make the CA appended. Redhat decided to trust an internal (tls) secret to the cluster for their operation, they did an error assuming the CA will always be present in the certificate chain, but if you can't trust the internal objects, where the automation can start? Should I remember here it WAS working prior to cert-manager 1.3 or something like that, it means it's a regression that must be solved imho.

charles-horel · 2021-08-31T14:38:27Z

This idea of not presenting the root is baked into the TLS standard. I’d say that the logical direction is for Redhat to provide a mechanism to trust the root CA. If it is using TLS properly that should be all that is required to resolve the operator degraded state for authentication. I haven’t played around with the cluster trust store, trusting it via machine config, or worst case (RFE to improve authentication to trust custom root CAs), but if I do get time to plug away at it more that will be my next step. Charles Horel, P.Eng E: ***@***.******@***.***> From: absynth76 ***@***.***> Sent: Tuesday, August 31, 2021 12:03 AM To: jetstack/cert-manager ***@***.***> Cc: Charles Horel ***@***.***>; Mention ***@***.***> Subject: Re: [jetstack/cert-manager] OCP Ingress Support for Cert-Manager 1.4 (#4247) CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. And again I'll repeat myself, MAY goes in two directions, it means can or cannot, it means a flag should be implemented to make the CA appended. Redhat decided to trust an internal (tls) secret to the cluster for their operation, they did an error assuming the CA will always be present in the certificate chain, but if you can't trust the internal objects, where the automation can start? Should I remember here it WAS working prior to cert-manager 1.3 or something like that, it means it's a regression that must be solved imho. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4247 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU6LYPOOYKT33TK5TFVWZPLT7RWA7ANCNFSM5A4DZ6NA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

absynth76 · 2021-08-31T19:27:54Z

Don't waste your time, I did test via a machine config assuming if the CA is in the host trust store it should work. It did not. It means redhat use only that tls.crt as a trust store for that specific flow as far as I understand.

jetstack-bot · 2021-11-29T20:08:22Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

jetstack-bot · 2021-12-29T20:28:25Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

jetstack-bot · 2022-01-28T20:34:25Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

jetstack-bot · 2022-01-28T20:34:27Z

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 23, 2021

jetstack-bot added the triage/support Indicates an issue that is a support question. label Jul 23, 2021

SgtCoDFish mentioned this issue Aug 19, 2021

Add a FullCertChain flag #4391

Closed

jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 29, 2021

jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 29, 2021

jetstack-bot closed this as completed Jan 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCP Ingress Support for Cert-Manager 1.4 #4247

OCP Ingress Support for Cert-Manager 1.4 #4247

charles-horel commented Jul 23, 2021

jakexks commented Jul 23, 2021 •

edited

jakexks commented Jul 23, 2021

charles-horel commented Jul 23, 2021 via email

charles-horel commented Jul 28, 2021

absynth76 commented Aug 27, 2021 •

edited

charles-horel commented Aug 30, 2021 •

edited

absynth76 commented Aug 31, 2021

charles-horel commented Aug 31, 2021 via email

absynth76 commented Aug 31, 2021

jetstack-bot commented Nov 29, 2021

jetstack-bot commented Dec 29, 2021

jetstack-bot commented Jan 28, 2022

jetstack-bot commented Jan 28, 2022

OCP Ingress Support for Cert-Manager 1.4 #4247

OCP Ingress Support for Cert-Manager 1.4 #4247

Comments

charles-horel commented Jul 23, 2021

jakexks commented Jul 23, 2021 • edited

jakexks commented Jul 23, 2021

charles-horel commented Jul 23, 2021 via email

charles-horel commented Jul 28, 2021

absynth76 commented Aug 27, 2021 • edited

charles-horel commented Aug 30, 2021 • edited

absynth76 commented Aug 31, 2021

charles-horel commented Aug 31, 2021 via email

absynth76 commented Aug 31, 2021

jetstack-bot commented Nov 29, 2021

jetstack-bot commented Dec 29, 2021

jetstack-bot commented Jan 28, 2022

jetstack-bot commented Jan 28, 2022

jakexks commented Jul 23, 2021 •

edited

absynth76 commented Aug 27, 2021 •

edited

charles-horel commented Aug 30, 2021 •

edited