[helm] Eclipse Che fail to start when using self-signed-cert on k8s #16429

elavicount · 2020-03-24T04:24:19Z

Describe the bug

I'm trying to deploy Eclipse Che on a Kubernetes cluster using self-signed-cert and the helm installer, after following the doc the problem persists.

Che version

latest
nightly
other: please specify

Steps to reproduce

Create a namespace:
kubectl create namespace che
Set environment variables:

/etc/profile.d/che.sh

export CA_CN=eclipse-che-signer
export DOMAIN="*.10.96.0.1.nip.io"
export OPENSSL_CNF=/etc/pki/tls/openssl.cnf

Generate self signed certificate:

#!/bin/bash
echo "CA_CN: ${CA_CN}"
echo "Domain: ${DOMAIN}"
echo "OPENSSL_CNF: ${OPENSSL_CNF}"

openssl genrsa -out rootCA.key 4096 &&  \
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt -subj /CN=${CA_CN} -reqexts SAN -extensions SAN -config <(cat ${OPENSSL_CNF} <(printf '[SAN]\nbasicConstraints=critical, CA:TRUE\nkeyUsage=keyCertSign, cRLSign, digitalSignature, keyEncipherment')) && \
openssl genrsa -out domain.key 2048 && \
openssl req -new -sha256 -key domain.key -subj "/O=EclipseChe/CN=${DOMAIN}" -reqexts SAN -config <(cat ${OPENSSL_CNF} <(printf "\n[SAN]\nsubjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth")) -out domain.csr && \
openssl x509 -req -sha256 -extfile <(printf "subjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth") -days 365 -in domain.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out domain.crt
cp rootCA.crt ca.crt

Create the secret key:

#!/bin/bash
kubectl delete secret che-tls -n che
kubectl create secret tls che-tls --key=domain.key --cert=domain.crt -n che
kubectl create secret generic self-signed-certificate --from-file=ca.crt -n che

Execute the following command in a kubernetes cluster:

chectl server:start --platform=k8s --multiuser --domain=10.96.0.1.nip.io --installer=helm --postgres-pvc-storage-class-name=local-storage --chenamespace=che --tls --self-signed-cert

Wait until che pod fails, this cause a CrashLoopBackOff

Expected behavior

Eclipse che should be deployed correctly

Runtime

kubernetes

Kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Screenshots

Installation method

chectl

chectl version
chectl/7.10.0 linux-x64 node-v10.19.0

helm version
version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

che-operator
minishift-addon
I don't know

Environment

Eclipse Che Logs

Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.10.96.0.1.nip.io/auth/realms/che/.well-known/openid-configuration
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:104)
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings$$FastClassByGuice$$e0d0786b.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.RealMapBinder$RealMapProvider.doProvision(RealMapBinder.java:796)
        at com.google.inject.internal.RealMapBinder$RealMapProvider.doProvision(RealMapBinder.java:733)
        at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:140)
        at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
        at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:85)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1841)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching keycloak-che.10.96.0.1.nip.io found.
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at java.net.URL.openStream(URL.java:1067)
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:97)
        ... 86 more

Additional context

kubectl get all -n che -o wide
NAME                                    READY   STATUS             RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
pod/che-6595df56b8-rx72n                0/1     CrashLoopBackOff   9          37m   10.244.2.54   node-2   <none>           <none>
pod/devfile-registry-696845fc9f-2zhrb   1/1     Running            0          37m   10.244.2.53   node-2   <none>           <none>
pod/keycloak-5cdcb5c9f6-pq2pg           1/1     Running            0          37m   10.244.1.79   node-1   <none>           <none>
pod/plugin-registry-7b94d46db7-xzrh8    1/1     Running            0          37m   10.244.2.55   node-2   <none>           <none>
pod/postgres-7f7bbd9c6d-qdxf4           1/1     Running            0          37m   10.244.1.80   node-1   <none>           <none>

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE   SELECTOR
service/che-host           ClusterIP   10.109.12.206    <none>        8080/TCP,8087/TCP   37m   app=che,component=che
service/devfile-registry   ClusterIP   10.104.173.178   <none>        8080/TCP            37m   app=che,component=devfile-registry
service/keycloak           ClusterIP   10.97.216.36     <none>        5050/TCP            37m   io.kompose.service=keycloak
service/plugin-registry    ClusterIP   10.109.123.99    <none>        8080/TCP            37m   app=che,component=plugin-registry
service/postgres           ClusterIP   10.110.91.235    <none>        5432/TCP            37m   io.kompose.service=postgres

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS             IMAGES                                        SELECTOR
deployment.apps/che                0/1     1            0           37m   che                    quay.io/eclipse/che-server:7.10.0             app=che
deployment.apps/devfile-registry   1/1     1            1           37m   che-devfile-registry   quay.io/eclipse/che-devfile-registry:7.10.0   app=che,component=devfile-registry
deployment.apps/keycloak           1/1     1            1           37m   keycloak               quay.io/eclipse/che-keycloak:7.10.0           app=che,component=keycloak
deployment.apps/plugin-registry    1/1     1            1           37m   che-plugin-registry    quay.io/eclipse/che-plugin-registry:7.10.0    app=che,component=plugin-registry
deployment.apps/postgres           1/1     1            1           37m   postgres               quay.io/eclipse/che-postgres:7.10.0           app=che,component=postgres

NAME                                          DESIRED   CURRENT   READY   AGE   CONTAINERS             IMAGES                                        SELECTOR
replicaset.apps/che-6595df56b8                1         1         0       37m   che                    quay.io/eclipse/che-server:7.10.0             app=che,pod-template-hash=6595df56b8
replicaset.apps/devfile-registry-696845fc9f   1         1         1       37m   che-devfile-registry   quay.io/eclipse/che-devfile-registry:7.10.0   app=che,component=devfile-registry,pod-template-hash=696845fc9f
replicaset.apps/keycloak-5cdcb5c9f6           1         1         1       37m   keycloak               quay.io/eclipse/che-keycloak:7.10.0           app=che,component=keycloak,pod-template-hash=5cdcb5c9f6
replicaset.apps/plugin-registry-7b94d46db7    1         1         1       37m   che-plugin-registry    quay.io/eclipse/che-plugin-registry:7.10.0    app=che,component=plugin-registry,pod-template-hash=7b94d46db7
replicaset.apps/postgres-7f7bbd9c6d           1         1         1       37m   postgres               quay.io/eclipse/che-postgres:7.10.0           app=che,component=postgres,pod-template-hash=7f7bbd9c6d

 kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3d23h

kubectl get ingress -n che
NAME               HOSTS                                   ADDRESS   PORTS     AGE
che-ingress        che-che.10.96.0.1.nip.io                          80, 443   45m
devfile-registry   devfile-registry-che.10.96.0.1.nip.io             80, 443   45m
keycloak-ingress   keycloak-che.10.96.0.1.nip.io                     80, 443   45m
plugin-registry    plugin-registry-che.10.96.0.1.nip.io              80, 443   45m

Possible duplicates #16396, #16280

Also could you please tell if there is any way to use a non nightly version, I need a stable version for production enviroment.

The text was updated successfully, but these errors were encountered:

tolusha · 2020-03-24T13:50:00Z

@eder-santos
The following doc [1] is relevant when you use operator installer.
In case of helm there are no manual steps. Just run the command below. tls will be enabled by default and self-sign certificate will be generated.

chectl server:start --platform=k8s --multiuser --domain=10.96.0.1.nip.io --installer=helm --postgres-pvc-storage-class-name=local-storage --chenamespace=che

[1] https://www.eclipse.org/che/docs/che-7/setup-che-in-tls-mode-with-self-signed-certificate/#deploy-che-with-self-signed-tls-on-kubernetes_setup-che-in-tls-mode-with-self-signed-certificate

elavicount · 2020-03-24T14:55:41Z

Hi @tolusha
I delete the whole namespace in order to destroy all resources and try again as with the command that you provide me, but unfortunately I got the same exception.

Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.10.96.0.1.nip.io/auth/realms/che/.well-known/openid-configuration
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:104)
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings$$FastClassByGuice$$e0d0786b.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:52)
        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:140)
        at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
        at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:85)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1841)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching keycloak-che.10.96.0.1.nip.io found.
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at java.net.URL.openStream(URL.java:1067)
        at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:97)
        ... 50 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching keycloak-che.10.96.0.1.nip.io found.
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:428)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:209)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
        ... 64 more

These are the secrets generated by helm:

[root@master-node scripts]# kubectl get secret -n che
NAME                        TYPE                                  DATA   AGE
che-keycloak-token-gdwmf    kubernetes.io/service-account-token   3      5m46s
che-tls                     kubernetes.io/tls                     3      5m56s
che-token-dgxzl             kubernetes.io/service-account-token   3      5m46s
default-token-wpbzh         kubernetes.io/service-account-token   3      6m35s
sh.helm.release.v1.che.v1   helm.sh/release.v1                    1      5m46s
[root@master-node scripts]# kubectl describe secret che-tls -n che
Name:         che-tls
Namespace:    che
Labels:       <none>
Annotations:  cert-manager.io/alt-names: 10.96.0.1.nip.io,*.10.96.0.1.nip.io
              cert-manager.io/certificate-name: che-certificate
              cert-manager.io/common-name: *.10.96.0.1.nip.io
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: che-cluster-issuer
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
ca.crt:   1854 bytes
tls.crt:  1586 bytes
tls.key:  1679 bytes
[root@master-node scripts]# kubectl describe secret che-keycloak-token-gdwmf -n che
Name:         che-keycloak-token-gdwmf
Namespace:    che
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: che-keycloak
              kubernetes.io/service-account.uid: 3d802c5a-8b48-4a2f-bfc6-5227b67a8bbe

Type:  kubernetes.io/service-account-token

Data
====
namespace:  3 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkhpbllIM2VTN1VFVUQzaE1uYzhoQTRHXzcwemtaVWZBWWgzV1NteFVFSTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjaGUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2hlLWtleWNsb2FrLXRva2VuLWdkd21mIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNoZS1rZXljbG9hayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNkODAyYzVhLThiNDgtNGEyZi1iZmM2LTUyMjdiNjdhOGJiZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjaGU6Y2hlLWtleWNsb2FrIn0.nuSKpA091q-YHVpwqnTFu56BtMbKCuRK1HcrO9jwCsgTmZ9r5JluNU3Wcabd9Aiqv1Z3dU6zpfodMdu_LCJeZeep2aEm1Dj-98K1JFBclbDkUG-NjgAnUme0NXlj_vT4zQQvF0XGaWcsdT3T8edbYnIijDplb1at6Qx32yWvN4WfSsXlY7UyAWzWlul9Qh1n09HDGvAmfVjhS6b_E4TCVneK5C725CXGHfLruQLx3zbhEp0mZKwwL-amlTeHK0kIiGr0utat2olcZodrHZuu7szaHXZf8elhJqJ-lWNOdjvvdPtar3atK5L8hkJwA-NpfBm4_7QcNXtZ3nyqdXhTng
ca.crt:     1025 bytes

tolusha · 2020-03-25T12:52:08Z

@eder-santos
kubectl get configmap che -n che -o=yaml | grep CHE_INFRA_KUBERNETES_TLS__SECRET

tolusha · 2020-03-25T13:53:50Z

Also
kubectl describe ingress keycloak-ingress -n che

elavicount · 2020-03-25T14:11:38Z

Thanks @tolusha
Here is the additional data, let me know if you need more information. Thanks again

[root@master-node chectl-logs]# kubectl get configmap che -n che -o=yaml | grep CHE_INFRA_KUBERNETES_TLS__SECRET
  CHE_INFRA_KUBERNETES_TLS__SECRET: che-tls

[root@master-node chectl-logs]# kubectl describe ingress keycloak-ingress -n che
Name:             keycloak-ingress
Namespace:        che
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  che-tls terminates keycloak-che.10.96.0.1.nip.io
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  keycloak-che.10.96.0.1.nip.io
                                 /   keycloak:5050 (10.244.1.99:8080)
Annotations:
  kubernetes.io/ingress.class:                        nginx
  nginx.ingress.kubernetes.io/proxy-connect-timeout:  3600
  nginx.ingress.kubernetes.io/proxy-read-timeout:     3600
  nginx.ingress.kubernetes.io/ssl-redirect:           true
Events:

tolusha · 2020-03-25T14:41:40Z

kubectl get deployment che -n che -o=yaml

elavicount · 2020-03-25T15:11:24Z

Thanks @tolusha, feel free to ask me any question.

kubectl get deployment che -n che -o=yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2020-03-25T15:09:29Z"
  generation: 1
  labels:
    app: che
    component: che
  name: che
  namespace: che
  resourceVersion: "1296122"
  selfLink: /apis/apps/v1/namespaces/che/deployments/che
  uid: 7445efd4-241c-4fab-a041-9ba898db8cb2
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: che
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: che
        component: che
    spec:
      containers:
      - env:
        - name: OPENSHIFT_KUBE_PING_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CHE_INFRA_KUBERNETES_TLS__CERT
          valueFrom:
            secretKeyRef:
              key: tls.crt
              name: che-tls
              optional: false
        - name: CHE_INFRA_KUBERNETES_TLS__KEY
          valueFrom:
            secretKeyRef:
              key: tls.key
              name: che-tls
              optional: false
        envFrom:
        - configMapRef:
            name: che
        image: quay.io/eclipse/che-server:7.10.0
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/system/state
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 120
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        name: che
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 8000
          name: http-debug
          protocol: TCP
        - containerPort: 8888
          name: jgroups-ping
          protocol: TCP
        - containerPort: 8087
          name: http-metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/system/state
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 60
        resources:
          limits:
            memory: 600Mi
          requests:
            memory: 256Mi
        securityContext:
          runAsUser: 1724
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      initContainers:
      - env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ENDPOINT
          value: postgres
        image: quay.io/eclipse/che-endpoint-watcher:7.10.0
        imagePullPolicy: IfNotPresent
        name: wait-for-postgres
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ENDPOINT
          value: keycloak
        image: quay.io/eclipse/che-endpoint-watcher:7.10.0
        imagePullPolicy: IfNotPresent
        name: wait-for-keycloak
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1724
      serviceAccount: che
      serviceAccountName: che
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastTransitionTime: "2020-03-25T15:09:29Z"
    lastUpdateTime: "2020-03-25T15:09:29Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2020-03-25T15:09:29Z"
    lastUpdateTime: "2020-03-25T15:09:29Z"
    message: ReplicaSet "che-8bf974698" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

tolusha · 2020-03-25T15:22:11Z

I see
you don't have env var. But how you get it

 - name: CHE_SELF__SIGNED__CERT
          valueFrom:
            secretKeyRef:
              key: ca.crt
              name: che-tls
              optional: false

tolusha · 2020-03-25T15:25:35Z

Did you use --self-signed-cert flag ?

tolusha · 2020-03-25T15:37:48Z

So root cause is that before deploying Eclipse Che the che-tls secret existed in namespace che.

tolusha · 2020-03-25T15:39:11Z

I assume that che namespace was not vanished properly before second installation

elavicount · 2020-03-25T16:12:22Z

Hi @tolusha
I remove the whole namespace the following way, but I'm still getting the same error.

kubectl delete secret che-tls -n che

kubectl delete secret self-signed-certificate -n che

kubectl delete secret self-signed-cert -n che

kubectl get pvc -n che | awk 'NR > 1 { print $1 }' | xargs -L 1 kubectl delete pvc $1 -n che

kubectl get pv -n che | awk 'NR > 1 { print $1 }' | xargs -L 1 kubectl delete pv $1 -n che

kubectl get storageclass -n che | awk 'NR > 1 { print $1 }' | xargs -L 1 kubectl delete storageclass $1 -n che

kubectl delete all --all -n che

kubectl delete namespace che

and then create again the namespace
kubectl create namespace che

I'm not using the --self-signed-cert option, but I notice the env var is created even without that option. This is my start command

chectl server:start --platform=k8s --multiuser --domain=10.96.0.1.nip.io --installer=helm --postgres-pvc-storage-class-name=local-storage --chenamespace=che

Let me know if you need more information, thank you

tolusha · 2020-03-26T07:37:50Z

@eder-santos
I will try to reproduce locally.

tolusha · 2020-03-26T08:51:43Z

Meanwhile you can try to deploy on minikube [1] cluster.
[1] https://kubernetes.io/docs/tasks/tools/install-minikube/

elavicount · 2020-03-26T14:50:15Z

Thank you @tolusha I will try it

mmorhun · 2020-04-03T10:07:42Z

@eder-santos have you succeeded with any of the installation methods?

elavicount · 2020-04-03T14:47:29Z

Hi @mmorhun
I try to run it with minikube but I'm unable to make it run properly in a CentOS Virtual Machine as I can't use the Kubernetes Cluster to install minikube. We are currently evaluating Eclipse Che as PaaS as we are unable to solve this problem running it on the cluster.

mmorhun · 2020-04-06T07:56:09Z

@eder-santos Minikube installation is designed to try it on single machine or for development purposes. The same I may say about CRC, but there are some tries to make it accessible outside. You may take a look at this one.

Can you try to deploy Che as you described in the issue description but using --installer=operator ?

tolusha · 2020-04-24T14:29:38Z

@eder-santos
chectl version 7.10.0 was used.
It contains several bugs related to deploying che with tls.
pls update it and retry again
chectl update stable

tolusha · 2020-04-28T07:13:36Z

@eder-santos
I close this issue since it was cause outdated version of chectl.

elavicount added the kind/bug Outline of a bug - must adhere to the bug report template. label Mar 24, 2020

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Mar 24, 2020

sleshchenko added the area/install Issues related to installation, including offline/air gap and initial setup label Mar 24, 2020

tolusha added the status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. label Mar 25, 2020

tolusha added severity/P2 Has a minor but important impact to the usage or development of the system. and removed status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. labels Mar 26, 2020

tolusha changed the title ~~Eclipse Che fail to start when using self-signed-cert~~ [helm] Eclipse Che fail to start when using self-signed-cert on k8s Mar 26, 2020

tolusha mentioned this issue Apr 13, 2020

Time out error when downloading images #16545

Closed

17 tasks

tolusha mentioned this issue Apr 21, 2020

Che Deploy Sprint #183 #16623

Closed

50 tasks

tolusha closed this as completed Apr 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[helm] Eclipse Che fail to start when using self-signed-cert on k8s #16429

[helm] Eclipse Che fail to start when using self-signed-cert on k8s #16429

elavicount commented Mar 24, 2020

tolusha commented Mar 24, 2020

elavicount commented Mar 24, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020 •

edited

Loading

elavicount commented Mar 25, 2020

tolusha commented Mar 25, 2020

elavicount commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020 •

edited

Loading

elavicount commented Mar 25, 2020

tolusha commented Mar 26, 2020

tolusha commented Mar 26, 2020

elavicount commented Mar 26, 2020

mmorhun commented Apr 3, 2020

elavicount commented Apr 3, 2020

mmorhun commented Apr 6, 2020

tolusha commented Apr 24, 2020

tolusha commented Apr 28, 2020

[helm] Eclipse Che fail to start when using self-signed-cert on k8s #16429

[helm] Eclipse Che fail to start when using self-signed-cert on k8s #16429

Comments

elavicount commented Mar 24, 2020

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Eclipse Che Logs

Additional context

tolusha commented Mar 24, 2020

elavicount commented Mar 24, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020 • edited Loading

elavicount commented Mar 25, 2020

tolusha commented Mar 25, 2020

elavicount commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020

tolusha commented Mar 25, 2020 • edited Loading

elavicount commented Mar 25, 2020

tolusha commented Mar 26, 2020

tolusha commented Mar 26, 2020

elavicount commented Mar 26, 2020

mmorhun commented Apr 3, 2020

elavicount commented Apr 3, 2020

mmorhun commented Apr 6, 2020

tolusha commented Apr 24, 2020

tolusha commented Apr 28, 2020

tolusha commented Mar 25, 2020 •

edited

Loading

tolusha commented Mar 25, 2020 •

edited

Loading