Error: Failed to run the workspace: "Server 'theia' in container 'theia-idej18' not available."

[peidong-hu]

Describe the bug

Deployed the Eclipse Che 7.1.0 in baremetal k8s v1.11, open dashboard correctly, but when creating workspace, it failed at error 'Error: Failed to run the workspace: "Server 'theia' in container 'theia-idej18' not available.'

Che version

7.1.0

• latest
• nightly
• other: please specify

Steps to reproduce

1. deploy to k8s v1.11 baremetal cluster
2. open che dashboard to create workspace
3. everything works fine until the code runs at here, https://github.com/eclipse/che/blob/22c2385cee95a3c59db2e1337ee402136f034465/wsmaster/che-core-api-workspace/src/main/java/org/eclipse/che/api/workspace/server/hc/ServerChecker.java#L123

which raised the error 'Error: Failed to run the workspace: "Server 'theia' in container 'theia-idej18' not available."'

or code at here, https://github.com/eclipse/che/blob/22c2385cee95a3c59db2e1337ee402136f034465/wsmaster/che-core-api-workspace/src/main/java/org/eclipse/che/api/workspace/server/hc/ServerChecker.java#L80

those are the only two places which print that error.

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

helm install -f ./values.yaml

• chectl
• che-operator
• minishift-addon
• I don't know

Environment

baremetal k8s cluster v 1.11

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

[peidong-hu]

I also tried che 6.19.0, it works with java stack, but failed at stack '
Java Theia on Kubernetes
Che development + Theia in a sidecar container (kubernetes)' when creating workspace.
The error message shows that wrong tag name for theia image

2019-09-12 00:36:52,001[nio-8080-exec-1]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 511]    - Workspace 'che/k8s-sidecar' with id 'workspacegaeucvhzjf3fjiba' created by user 'che'
2019-09-12 00:36:52,912[nio-8080-exec-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 383]   - Starting workspace 'che/k8s-sidecar' with id 'workspacegaeucvhzjf3fjiba' by user 'che'
2019-09-12 00:37:16,657[/10.233.0.1/...]  [ERROR] [.i.k.KubernetesInternalRuntime 915]  - Unrecoverable event occurred during workspace 'workspacegaeucvhzjf3fjiba' startup: Failed, Failed to pull image "eclipse/che-theia:0.3.19-6.19.0": rpc error: code = Unknown desc = Error response from daemon: manifest for eclipse/che-theia:0.3.19-6.19.0 not found, workspacegaeucvhzjf3fjiba.ws-56f5577b7-fzqnt
2019-09-12 00:37:16,660[aceSharedPool-1]  [WARN ] [.i.k.KubernetesInternalRuntime 243]  - Failed to start Kubernetes runtime of workspace workspacegaeucvhzjf3fjiba. Cause: Unrecoverable event occurred: 'Failed', 'Failed to pull image "eclipse/che-theia:0.3.19-6.19.0": rpc error: code = Unknown desc = Error response from daemon: manifest for eclipse/che-theia:0.3.19-6.19.0 not found', 'workspacegaeucvhzjf3fjiba.ws-56f5577b7-fzqnt'
2019-09-12 00:37:17,249[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 803]   - Workspace 'che:k8s-sidecar' with id 'workspacegaeucvhzjf3fjiba' start failed
2019-09-12 00:38:26,548[nio-8080-exec-6]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 383]   - Starting workspace 'che/k8s-sidecar' with id 'workspacegaeucvhzjf3fjiba' by user 'che'

[peidong-hu]

It seems that both version 7.1.0 and 6.19.0 have some bugs in theia module. not sure if they have the same cause though.

[skabashnyuk]

Hello @peidong-hu . What was your installation method and what kind of Cloud provider did you use for your k8s?

[peidong-hu]

my k8s runs on baremetal in our lab. loadbalancer is metallb with externalDNS automate the dns entry creation.

I used the helm installed it.

[tsmaeder]

Classifying as a P1 bug: workspaces generally not starting is bed. However, it seems to be restricted to a particular setup.

[skabashnyuk]

@peidong-hu I don't know how to reproduce your issue. Can you try to install che with chectl?

[huangli812]

I met the same problem, I sovled it by config the DNS server of my kubernetes cluster,so that other pod can access to the new created eclipse theia server service. I thought it may have same problem with your DNS, loadbalancer or domain setting.

[peidong-hu]

Hi Huangli, I am using metallb as load balancer, external DNS ( https://github.com/kubernetes-incubator/external-dns) and Bind9 as DNS server to automatically create DNS entry. For other deployment, all the DNS entries work well. What DNS , load balancer solution do u use? Thanks Peidong

On Sat, Sep 28, 2019, 5:43 AM huangli812 ***@***.***> wrote: I met the same problem, I sovled it by config the DNS server of my kubernetes cluster,so that other pod can access to the new created eclipse theia server service. I thought it may have same problem with your DNS, loadbalancer or domain setting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14517>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2LBC4ZWGVOFAWE22LOEJLQL4YNHANCNFSM4IV3KQCA> .

On Sep 28, 2019 5:43 AM, "huangli812" <notifications@github.com> wrote: I met the same problem, I sovled it by config the DNS server of my kubernetes cluster,so that other pod can access to the new created eclipse theia server service. I thought it may have same problem with your DNS, loadbalancer or domain setting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14517>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2LBC4ZWGVOFAWE22LOEJLQL4YNHANCNFSM4IV3KQCA> .

[maciejkozuch]

Hi,
I've had the same issue. I've configured the version 7 of Che on the OVH Kubernetes witch chectl tool with the following command:

chectl server:start -b mydomain.com -m -p k8s -s

Before, I had created DNS records for che-che.mydomain.com, keycloak-che.mydomain.com , plugin-registry-che.mydomain.com and devfile-registry-che.mydomain.com in my DNS server.

I've solved the problem by create a *.che.mydomain.com record in my DNS server and run chectl with -b che.mydomain.com.

[peidong-hu]

That's interesting, so what IP address is your *.che.mydomain.com pointing to? In my case, all domains are automatically created during the che deployment, which seem all working well. It will be a big mystery about the *.che.mydomain.com in your case, how it could help?

…

On Mon, Sep 30, 2019 at 10:57 AM maciejkozuch ***@***.***> wrote: Hi, I've had the same issue. I've configured the version 7 of Che on the OVH Kubernetes witch chectl tool with the following command: chectl server:start -b mydomain.com -m -p k8s -s Before, I had created DNS records for che-che.mydomain.com, keycloak-che.mydomain.com , plugin-registry-che.mydomain.com and devfile-registry-che.mydomain.com in my DNS server. I've solved the problem by create a *.che.mydomain.com record in my DNS server and run chectl with -b che.mydomain.com. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14517>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2LBCYG4F3FSUCMEHWFO4TQMIOU5ANCNFSM4IV3KQCA> .

[peidong-hu]

This issue just disappears when I deploy the latest 7.2.0 version. I think we may close it at this moment.

[andy316x]

Fixed by #14052

[caiziqi33]

I met the same problem, I sovled it by config the DNS server of my kubernetes cluster,so that other pod can access to the new created eclipse theia server service. I thought it may have same problem with your DNS, loadbalancer or domain setting.

I also encountered the same problem：Error: Failed to run the workspace: "Server 'theia' in container 'theia-idegoz' not available."
The environment is built with reference to the official website doc minikube method，Can you specifically talk about the configuration of codedns，thankyou

[caiziqi33]

Hi,
I've had the same issue. I've configured the version 7 of Che on the OVH Kubernetes witch chectl tool with the following command:

chectl server:start -b mydomain.com -m -p k8s -s

Before, I had created DNS records for che-che.mydomain.com, keycloak-che.mydomain.com , plugin-registry-che.mydomain.com and devfile-registry-che.mydomain.com in my DNS server.

I've solved the problem by create a *.che.mydomain.com record in my DNS server and run chectl with -b che.mydomain.com.

I also encountered the same problem：Error: Failed to run the workspace: "Server 'theia' in container 'theia-idegoz' not available."
The environment is built with reference to the official website doc minikube method，@maciejkozuch Can you specifically talk about the configuration of codedns，thankyou

[m-hu]

This may still happen due to other unknown kind of http IOException, as the exception has been handled but the exception is relatively broad in such a context, which could correspond to other no-bug cases.