Repository navigation
Closed
Labels
Description
Describe the bug
Gitpod cannot initial workspace with the following error
cannot initialize workspace: cannot connect to ws-sync: cannot connect to ws-sync: cannot connect to workspace sync
Further investigation of the logs of the deployed workspace show the Theia server immediately shutting down
{"component":"workspace","severity":"info","time":"2020-10-19T19:59:54.249Z","message":"Registered built-in plugin","payload":{"name":"root","params":[{"name":"vscode.docker@1.47.3","loc":"/theia/node_modules/@gitpod/gitpod-ide/plugins/vscode.docker-1.47.3.vsix"}]}}
{"message":"Theia supervisor loop ended - waiting for Theia to come down","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"}
{"message":"shutting down health endpoint","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"}
{"error":null,"message":"Theia was stopped","serviceContext":{"service":"theia-supervisor","version":""},"severity":"warning","time":"2020-10-19T19:59:58Z"}
What is interesting is this error does not occur 100% of the time. Applying changes to the helm chart causing certain pods to redeploy will allow it to work. It does not occur when using a single-node K8s cluster, so perhaps we have a scheduling issue where services are not talking across nodes correctly.
Steps to reproduce
- Deploy Gitpod to 2 or more Kubernetes nodes
- Apply changes to helm chart (sometimes this breaks it, sometimes this fixes it)
- If error exists, keep applying changes and/or redeploy one or more pods until services restore
Expected behavior
Workspace should initialize and the following should appear in the logs rather than the shutdown of Theia
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:38.586Z","message":"Using Git [2.26.0] from the PATH. (/usr/bin/git)","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning theia websocket reconnecting on","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning on theia websocket first time took: 1590.8 ms","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"warn","time":"2020-10-20T13:28:39.059Z","message":"Frontend t.configure is slow, took: 2176.6 ms","payload":{"name":"root","params":[]}}
Additional information
Red Hat Linux 7.8 (3.10.0-1127.el7.x86_64)
Docker CE 19.03.13
Rancher Kubernetes 17.5.1 (Calico networking)
Gitpod v0.5.0
Example repository
n/a
Activity
jgallucci32 commentedon Oct 20, 2020
It turns out it is a problem with the host networking configuration of the
ws-syncpod. I had to add the following to its manifest as a workaround to get around this problem:I am curious if other deployments see this or if this is specific to RKE due to its default pod security polies not allowing this.
inventionlabsSydney commentedon Apr 5, 2021
Hi, I'm experiencing this exact issue @jgallucci32
Could you please explain further how you implemented your fix? I'm struggling here.
I am unsure where to apply