cannot initialize workspace: cannot connect to ws-sync #2029
Description
Describe the bug
Gitpod cannot initial workspace with the following error
cannot initialize workspace: cannot connect to ws-sync: cannot connect to ws-sync: cannot connect to workspace sync
Further investigation of the logs of the deployed workspace show the Theia server immediately shutting down
{"component":"workspace","severity":"info","time":"2020-10-19T19:59:54.249Z","message":"Registered built-in plugin","payload":{"name":"root","params":[{"name":"vscode.docker@1.47.3","loc":"/theia/node_modules/@gitpod/gitpod-ide/plugins/vscode.docker-1.47.3.vsix"}]}}
{"message":"Theia supervisor loop ended - waiting for Theia to come down","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"}
{"message":"shutting down health endpoint","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"}
{"error":null,"message":"Theia was stopped","serviceContext":{"service":"theia-supervisor","version":""},"severity":"warning","time":"2020-10-19T19:59:58Z"}
What is interesting is this error does not occur 100% of the time. Applying changes to the helm chart causing certain pods to redeploy will allow it to work. It does not occur when using a single-node K8s cluster, so perhaps we have a scheduling issue where services are not talking across nodes correctly.
Steps to reproduce
- Deploy Gitpod to 2 or more Kubernetes nodes
- Apply changes to helm chart (sometimes this breaks it, sometimes this fixes it)
- If error exists, keep applying changes and/or redeploy one or more pods until services restore
Expected behavior
Workspace should initialize and the following should appear in the logs rather than the shutdown of Theia
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:38.586Z","message":"Using Git [2.26.0] from the PATH. (/usr/bin/git)","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning theia websocket reconnecting on","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning on theia websocket first time took: 1590.8 ms","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"warn","time":"2020-10-20T13:28:39.059Z","message":"Frontend t.configure is slow, took: 2176.6 ms","payload":{"name":"root","params":[]}}
Additional information
Red Hat Linux 7.8 (3.10.0-1127.el7.x86_64)
Docker CE 19.03.13
Rancher Kubernetes 17.5.1 (Calico networking)
Gitpod v0.5.0
Example repository
n/a