Use OpenShift service serving certificates to retrieve routes TLS CA

[l0rd]

Is your enhancement related to a problem? Please describe.

We currently generate our own tls certificates using a job and the image che tls secret creator.

This mechanism works fine for local installations but doesn't scale at the enterprise level where certificate management goes way beyond a simple certificate generation. When available (i.e. on OpenShift) we should delegate certificates management to an external service.

Describe the solution you'd like

When Che is deployed on OpenShift we should use its service serving certificates instead of the tls-secret-creator job.

[l0rd]

cc @mmorhun @raffaelespazzoli

[l0rd]

cc @davidfestal @sleshchenko

[mmorhun]

We do not use tls-secret-creator on Openshift infrastructures

[l0rd]

@mmorhun please provide more details about how do we generate certificates on OpenShift when using self-signed certificates.

[sleshchenko]

When Che is deployed on OpenShift we should use its service serving certificates instead of the tls-secret-creator job.

The own tls certificates using a job - generates certificates for public domain. I'm not sure that we can reuse it.
BTW Let's wait for Mykola's answer. I suppose on OpenShift we just rely on the default certificates that are set for routed by cluster administrator.

[l0rd]

The own tls certificates using a job - generates certificates for public domain. I'm not sure that we can reuse it.

@sleshchenko why?

[sleshchenko]

@l0rd The referenced che-operator code is invoked for K8s platform only https://github.com/eclipse/che-operator/blob/aee5ca35678a34fbaac7bf2f77c775aa5a282f9f/pkg/controller/che/che_controller.go#L363

why?

Because it's the certificate needed on Ingress level - mainly for securing the connection with client/browser.

[l0rd]

I suppose on OpenShift we just rely on the default certificates that are set for routed by cluster administrator.

The point here is how do we get the certificate of the CA that is used to sign the routes certificates? In the (outdated) documentation we suggested users to do that manually:

oc get secret router-ca -n openshift-ingress-operator -o jsonpath="{.data.tls\.crt}" | \
  base64 -d > ca.crt

But that command is not reliable. I doesn't work if the ingress controller in the cluster is not the default one. That's why we need to use service serving certificates or use the built-in certificate injection.

[mmorhun]

@l0rd we use tls-secret-creator only for Kubernetes family infrastructures. In case of Openshift we never generate certificates, but instead create a test route and read certificate chain from it (that's how it was before, we didn't change this part). In terms of automating self-signed certificate flag, I added functionality which analyzes the chain, so we know whether self-signed certificate is used and if so, we propagate it to some Che components.

[mmorhun]

As about getting CA certiificate, if it is self-signed, Openshift sends it with the chain as well, at least on the infrastructures I was able to test (CRC, clusterbot, QE OCP 4.x).

[l0rd]

@mmorhun understand. We do not generate certificates when running on OpenShift and we retrieve the CA cert from a fake route. However using OpenShift service serving certificate or certificates injection seems more reliable.

[mmorhun]

I don't think we should use ca-bandle injection as it will inject all trusted certificates of the cluster (a lot), because we need only one: to propagate to Che components and to give it to the users.

Unfortunately we cannot use service serving certificate as it creates certificates only for internal communications which is not going to work outside the cluster, i.e. Che end user cannot use them.

[l0rd]

I don't think we should use ca-bandle injection as it will inject all trusted certificates of the cluster (a lot), because we need only one: to propagate to Che components and to give it to the users.

Che should NOT manage certificates. Che doesn't know how to revoke a certificate, how to rotate it etc...We should delegate that to an external service. I only see benefits of having (a lot) of trusted certificates injected via service serving.

Unfortunately we cannot use service serving certificate as it creates certificates only for internal communications which is not going to work outside the cluster, i.e. Che end user cannot use them.

Why do you think that a Che end user cannot use them? We already have documentation that explains how to import a certificate that is not trusted locally. What's the problem with that?

[mmorhun]

Che should NOT manage certificates. Che doesn't know how to revoke a certificate, how to rotate it etc...We should delegate that to an external service.

Completely agree, it is not Che responsibility. However, that adds requirements for the user (Che instance admin) to manually generate and apply certificates as we did before the TLS automation. It will not be a problem for correctly configured Openshift or Kubernetes (because that's already done on cluster setup), but certainly will add manual work for all other cases (including local ones like Minikube, CRC).

Why do you think that a Che end user cannot use them? We already have documentation that explains how to import a certificate that is not trusted locally. What's the problem with that?

The service serving certificates are bound to the internal cluster domain(s). No way to use it outside.

[l0rd]

The service serving certificates are bound to the internal cluster domain(s). No way to use it outside.

"Are bound to the internal cluster domain(s)"? I don't understand what you mean.

[l0rd]

Completely agree, it is not Che responsibility. However, that adds requirements for the user (Che instance admin) to manually generate and apply certificates as we did before the TLS automation. It will not be a problem for correctly configured Openshift or Kubernetes (because that's already done on cluster setup), but certainly will add manual work for all other cases (including local ones like Minikube, CRC).

Yes. We should use a certificate service when available (OCP) and generate certificates if we cannot really find a better solution.

[mmorhun]

"Are bound to the internal cluster domain(s)"? I don't understand what you mean.

I meant DNS constraints in subjectAltName of an x509 TLS certificate. From the docs of service serving certificates:

The generated certificate is only valid for the internal service DNS name <service.name>.<service.namespace>.svc, and are only valid for internal communications.

[l0rd]

@mmorhun we are talking about 2 different things:

1. Your link refers to the section of the documentation to add a certificate to a service
2. I am referring to the automatic injection of the CA bundle of the cluster (this section of the doc

As you mentioned correctly 1. is for internal hostnames. But this issue is about retrieving the cluster CA public certificates so that che-server, theia etc...trust it and so that users can copy it locally and configure their browser to trust it too. And we should leveage the automatic injection rather than extracting it from a test route. Does that make sense?

[mmorhun]

Sorry, I was confused by

certificates injected via service serving

and assumed using of service serving certificates. We indeed had different contexts. Thank you for pointing out.

this issue is about retrieving the cluster CA public certificates so that che-server, theia etc...trust it

+1 from me that Che (and all components like Che Theia, plugin broker, etc.) should trust configured in Openshift certificates (whole ca-bandle). It is clean way to be on the same position as the cluster.

so that users can copy it locally and configure their browser to trust it too

Do you mean CA certificates from the cluster ca-bandle?

[l0rd]

Do you mean CA certificates from the cluster ca-bandle?

We probably need only the certificate of the CA used to sign routes certificates (and that should always be in the trusted-ca-bundle).

[tolusha]

@l0rd
The doc https://docs.openshift.com/container-platform/4.4/security/certificates/service-serving-certificate.html#add-service-certificate-configmap_service-serving-certificate you referenced to allows to inject service CA certificate it is different from
ingress CA certificate :(

[tolusha]

How to get service CA certificate:

oc create configmap serviceca
oc annotate configmap serviceca service.beta.openshift.io/inject-cabundle=true
oc get configmap test -o yaml | yq -r  '.data["service-ca.crt"]'

How to get ingress CA certificate:

openssl s_client -connect console-openshift-console.<clusterdomain>:443   -showcerts

Decoding with openssl x509 -text -noout shows that service CA certificate is signed by openshift-service-serving-signer@1597906919 but ingress CA certificate is signed by ingress-operator@1597907234