Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly #17631

asavin-cl · 2020-08-13T12:30:20Z

When I deploy Che through chectl using olm installer and use CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username>
I got an error when try to start che workspace.
Error: Failed to run the workspace: "Server 'theia' in container 'theia-idemo8' not available."

The error relates to SSL issue.

I found the problem in ingress configuration.

tls:
- hosts:
- serverfz4halbv-jwtproxy-server-4400.192.168.64.109.nip.io
secretName: che-tls

But if I use chectl helm installer ingress configuration looks differently.
tls:
- hosts:
- server3d7rl40n-jwtproxy-server-4400.192.168.64.109.nip.io
secretName: workspace0kah0jezpmlppw58-che-tls

Therefore in olm installation case che is looking che-tls certificates in che-ws-<username> Kubernetes namespace but this secret absent there.

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minikube version: v1.12.2

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Screenshots

Installation method

chectl
- provide an output of chectl version command
  chectl/7.17.0 win32-x64 node-v10.22.0
- provide a full command that was used to deploy Eclipse Che (including the output)
  Relative to this issue need to Add Che service account.
  [che-operator] - serviceaccount permissions for <username>-che as default namespace #15493
  kubectl create clusterrolebinding che --clusterrole cluster-admin --serviceaccount=che:che
  Install Che

chectl server:start --installer olm --che-operator-cr-yaml=che.yaml --platform minikube --multiuser --k8spodreadytimeout=300000  --k8spodwaittimeout=600000

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
  name: eclipse-che
spec:
  server:
    gitSelfSignedCert: false
    selfSignedCert: true
    tlsSupport: true
    allowUserDefinedWorkspaceNamespaces: false
    customCheProperties:
      CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: 'che-ws-<username>'
      CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: "2"
  database:
    externalDb: false
  storage:
    pvcStrategy: 'per-workspace'
    pvcClaimSize: '2Gi'
    preCreateSubPaths: true
  auth:
    externalIdentityProvider: false
    openShiftoAuth: false
  k8s:
    ingressDomain: '192.168.64.111.nip.io'
    tlsSecretName: 'che-tls'
  metrics:
    enable: true

OperatorHub
I don't know

Environment

Che version 7.17.0

The text was updated successfully, but these errors were encountered:

tolusha · 2020-08-17T06:29:16Z

/cc @skabashnyuk

tolusha · 2020-08-17T06:35:08Z

@asavin-cl
nice catch!

skabashnyuk · 2020-08-17T07:23:17Z

We have such a properties in this case

# Name of a secret that should be used when creating workspace ingresses with TLS
# Ignored by OpenShift infrastructure
che.infra.kubernetes.tls_secret=

# Data for TLS Secret that should be used for workspaces Ingresses
# cert and key should be encoded with Base64 algorithm
# These properties are ignored by OpenShift infrastructure
che.infra.kubernetes.tls_key=NULL
che.infra.kubernetes.tls_cert=NULL

looks like with helm all 3 was configured and in case of olm only che.infra.kubernetes.tls_secret

skabashnyuk · 2020-08-17T07:23:37Z

I believe CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT is not related to tls issue.

asavin-cl · 2020-08-17T08:13:41Z

@skabashnyuk
There ConfigMaps from HELM installer as you can see che.infra.kubernetes.tls_key and che.infra.kubernetes.tls_cert not defined but workspaces are creating in different namespaces properly.

{
	"CHE_API": "https://che-che.192.168.64.111.nip.io/api",
	"CHE_CORS_ALLOWED__ORIGINS": "*",
	"CHE_CORS_ALLOW__CREDENTIALS": "false",
	"CHE_CORS_ENABLED": "false",
	"CHE_DEBUG_SERVER": "false",
	"CHE_HOST": "che-che.192.168.64.111.nip.io",
	"CHE_INFRASTRUCTURE_ACTIVE": "kubernetes",
	"CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL": "https://che-che.192.168.64.111.nip.io/agent-binaries/linux_amd64/bootstrapper/bootstrapper",
	"CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON": "{\"kubernetes.io/ingress.class\": \"nginx\", \"kubernetes.io/tls-acme\": \"true\", \"nginx.ingress.kubernetes.io/rewrite-target\": \"/$1\",\"nginx.ingress.kubernetes.io/ssl-redirect\": \"true\",\"nginx.ingress.kubernetes.io/proxy-connect-timeout\": \"3600\",\"nginx.ingress.kubernetes.io/proxy-read-timeout\": \"3600\"}",
	"CHE_INFRA_KUBERNETES_INGRESS_DOMAIN": "192.168.64.111.nip.io",
	"CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM": "%s(.*)",
	"CHE_INFRA_KUBERNETES_MASTER__URL": "",
	"CHE_INFRA_KUBERNETES_NAMESPACE": "che-ws-<username>",
	"CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT": "che-ws-<username>",
	"CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP": "1724",
	"CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER": "1724",
	"CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS": "true",
	"CHE_INFRA_KUBERNETES_PVC_QUANTITY": "1Gi",
	"CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME": "",
	"CHE_INFRA_KUBERNETES_PVC_STRATEGY": "per-workspace",
	"CHE_INFRA_KUBERNETES_SERVER__STRATEGY": "multi-host",
	"CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME": "che-workspace",
	"CHE_INFRA_KUBERNETES_TLS__ENABLED": "true",
	"CHE_INFRA_KUBERNETES_TLS__SECRET": "che-tls",
	"CHE_INFRA_KUBERNETES_TRUST__CERTS": "false",
	"CHE_INFRA_KUBERNETES_WORKSPACE__START__TIMEOUT__MIN": "15",
	"CHE_KEYCLOAK_AUTH__SERVER__URL": "https://keycloak-che.192.168.64.111.nip.io/auth",
	"CHE_KEYCLOAK_CLIENT__ID": "che-public",
	"CHE_KEYCLOAK_REALM": "che",
	"CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT": "1800000",
	"CHE_LOCAL_CONF_DIR": "/etc/conf",
	"CHE_LOGGER_CONFIG": "",
	"CHE_LOGS_APPENDERS_IMPL": "plaintext",
	"CHE_LOGS_DIR": "/data/logs",
	"CHE_LOG_LEVEL": "INFO",
	"CHE_METRICS_ENABLED": "false",
	"CHE_MULTIUSER": "true",
	"CHE_OAUTH_GITHUB_CLIENTID": "",
	"CHE_OAUTH_GITHUB_CLIENTSECRET": "",
	"CHE_OAUTH_OPENSHIFT_CLIENTID": "",
	"CHE_OAUTH_OPENSHIFT_CLIENTSECRET": "",
	"CHE_OAUTH_OPENSHIFT_OAUTH__ENDPOINT": "",
	"CHE_OAUTH_OPENSHIFT_VERIFY__TOKEN__URL": "",
	"CHE_PORT": "8080",
	"CHE_TRACING_ENABLED": "false",
	"CHE_WEBSOCKET_ENDPOINT": "wss://che-che.192.168.64.111.nip.io/api/websocket",
	"CHE_WORKSPACE_AUTO_START": "false",
	"CHE_WORKSPACE_DEVFILE__REGISTRY__URL": "https://devfile-registry-che.192.168.64.111.nip.io",
	"CHE_WORKSPACE_HTTPS__PROXY": "",
	"CHE_WORKSPACE_HTTP__PROXY": "",
	"CHE_WORKSPACE_JAVA__OPTIONS": "-Xmx2000m",
	"CHE_WORKSPACE_MAVEN__OPTIONS": "-Xmx20000m",
	"CHE_WORKSPACE_NO__PROXY": "",
	"CHE_WORKSPACE_PLUGIN__REGISTRY__URL": "https://plugin-registry-che.192.168.64.111.nip.io/v3",
	"JAEGER_ENDPOINT": "http://jaeger-collector:14268/api/traces",
	"JAEGER_REPORTER_MAX_QUEUE_SIZE": "10000",
	"JAEGER_SAMPLER_MANAGER_HOST_PORT": "jaeger:5778",
	"JAEGER_SAMPLER_PARAM": "1",
	"JAEGER_SAMPLER_TYPE": "const",
	"JAEGER_SERVICE_NAME": "che-server",
	"JAVA_OPTS": "-XX:MaxRAMPercentage=85.0 "
}

tolusha · 2020-08-17T13:03:09Z

@asavin-cl
I've updated the description. Pls, provide some more information by tagging appropriate items

asavin-cl · 2020-08-17T13:23:44Z

@tolusha
Information provided.
But I have the same problems with Google Kubernetes environment.

tolusha · 2020-08-18T09:10:40Z

After discussion @skabashnyuk we agreed that operator should set properties which are used to secure workspace ingresses (it is needed when workspace is created in another namespace).
The values of these properties must be the same as corresponding values of che-tls secret.

CHE_INFRA_KUBERNETES_TLS__KEY
CHE_INFRA_KUBERNETES_TLS__CERT

As workaround, pls create che-tls secret in a user namespace with the same content as che-tls secret in che namespace.

guydog28 · 2020-08-18T15:12:50Z

I can concur this problem exists for us too, using the chectl operator installer on vanilla k8s. copying the cert to the new generated namespace solved the issue as well, but I'm about to have a bunch of users sign up and create workspaces, and I will have to manually copy that secret for each one of them as the create. Any idea what release we could expect to see a fix? I'm going to report another bug with this arrangement in a moment, where you cannot deploy a custom devfile from the Che UI when the namespace is che-<workspaceid> (or any other placeholder) as it tries to validate the namespace name without substituting the placeholder, which is not a valid namespace name. See #17656

tolusha · 2020-08-19T05:57:53Z

The fix is expected to be in 7.19

tolusha · 2020-08-28T07:08:20Z

@mmorhun
A little hack to start workspaces.
kubectl create clusterrolebinding che --clusterrole cluster-admin --serviceaccount=che:che

mmorhun · 2020-08-31T11:24:17Z

Fixed.

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Aug 13, 2020

asavin-cl changed the title ~~The option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly when you use olm installer.~~ The option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly when use olm installer. Aug 13, 2020

rkratky assigned tolusha Aug 16, 2020

rkratky changed the title ~~The option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly when use olm installer.~~ Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly when using OLM installer Aug 16, 2020

rkratky added the kind/bug Outline of a bug - must adhere to the bug report template. label Aug 16, 2020

tolusha removed their assignment Aug 17, 2020

tolusha added area/che-server and removed area/install Issues related to installation, including offline/air gap and initial setup status/analyzing An issue has been proposed and it is currently being analyzed for effort and implementation approach labels Aug 17, 2020

tolusha added area/che-operator Issues and PRs related to Eclipse Che Kubernetes Operator and removed area/che-server labels Aug 18, 2020

tolusha added this to the Backlog - Deploy milestone Aug 18, 2020

guydog28 mentioned this issue Aug 18, 2020

Cannot create workspace from Custom devfile when using namespace placeholder like <workspaceid>-che #17656

Closed

22 tasks

tolusha changed the title ~~Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly when using OLM installer~~ Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly Aug 19, 2020

tolusha mentioned this issue Aug 19, 2020

Che Deploy Sprint #189 #17661

Closed

58 tasks

tolusha removed this from the Backlog - Deploy milestone Aug 26, 2020

tolusha added this to the 7.19 milestone Aug 26, 2020

mmorhun self-assigned this Aug 28, 2020

mmorhun mentioned this issue Aug 28, 2020

Add server TLS key and cert into Che configmap if a separate namespace for Che workspaces is configured eclipse-che/che-operator#421

Merged

mmorhun closed this as completed Aug 31, 2020

tolusha mentioned this issue Sep 1, 2020

Add more checks to tls certificates into che configmap eclipse-che/che-operator#425

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly #17631

Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly #17631

asavin-cl commented Aug 13, 2020 •

edited

Loading

tolusha commented Aug 17, 2020

tolusha commented Aug 17, 2020

skabashnyuk commented Aug 17, 2020

skabashnyuk commented Aug 17, 2020

asavin-cl commented Aug 17, 2020 •

edited

Loading

tolusha commented Aug 17, 2020

asavin-cl commented Aug 17, 2020 •

edited

Loading

tolusha commented Aug 18, 2020 •

edited

Loading

guydog28 commented Aug 18, 2020 •

edited

Loading

tolusha commented Aug 19, 2020

tolusha commented Aug 28, 2020

mmorhun commented Aug 31, 2020

Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly #17631

Option CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che-ws-<username> does not work properly #17631

Comments

asavin-cl commented Aug 13, 2020 • edited Loading

Runtime

Screenshots

Installation method

Environment

tolusha commented Aug 17, 2020

tolusha commented Aug 17, 2020

skabashnyuk commented Aug 17, 2020

skabashnyuk commented Aug 17, 2020

asavin-cl commented Aug 17, 2020 • edited Loading

tolusha commented Aug 17, 2020

asavin-cl commented Aug 17, 2020 • edited Loading

tolusha commented Aug 18, 2020 • edited Loading

guydog28 commented Aug 18, 2020 • edited Loading

tolusha commented Aug 19, 2020

tolusha commented Aug 28, 2020

mmorhun commented Aug 31, 2020

asavin-cl commented Aug 13, 2020 •

edited

Loading

asavin-cl commented Aug 17, 2020 •

edited

Loading

asavin-cl commented Aug 17, 2020 •

edited

Loading

tolusha commented Aug 18, 2020 •

edited

Loading

guydog28 commented Aug 18, 2020 •

edited

Loading