Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cannot initialize workspace: cannot connect to ws-sync #2029

Closed
jgallucci32 opened this issue Oct 20, 2020 · 2 comments
Closed

cannot initialize workspace: cannot connect to ws-sync #2029

jgallucci32 opened this issue Oct 20, 2020 · 2 comments

Comments

@jgallucci32
Copy link
Contributor

jgallucci32 commented Oct 20, 2020

Describe the bug

Gitpod cannot initial workspace with the following error

cannot initialize workspace: cannot connect to ws-sync: cannot connect to ws-sync: cannot connect to workspace sync

Further investigation of the logs of the deployed workspace show the Theia server immediately shutting down

{"component":"workspace","severity":"info","time":"2020-10-19T19:59:54.249Z","message":"Registered built-in plugin","payload":{"name":"root","params":[{"name":"vscode.docker@1.47.3","loc":"/theia/node_modules/@gitpod/gitpod-ide/plugins/vscode.docker-1.47.3.vsix"}]}} 
{"message":"Theia supervisor loop ended - waiting for Theia to come down","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"} 
{"message":"shutting down health endpoint","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"} 
{"error":null,"message":"Theia was stopped","serviceContext":{"service":"theia-supervisor","version":""},"severity":"warning","time":"2020-10-19T19:59:58Z"} 

What is interesting is this error does not occur 100% of the time. Applying changes to the helm chart causing certain pods to redeploy will allow it to work. It does not occur when using a single-node K8s cluster, so perhaps we have a scheduling issue where services are not talking across nodes correctly.

Steps to reproduce

  1. Deploy Gitpod to 2 or more Kubernetes nodes
  2. Apply changes to helm chart (sometimes this breaks it, sometimes this fixes it)
  3. If error exists, keep applying changes and/or redeploy one or more pods until services restore

Expected behavior

Workspace should initialize and the following should appear in the logs rather than the shutdown of Theia

{"component":"workspace","severity":"info","time":"2020-10-20T13:28:38.586Z","message":"Using Git [2.26.0] from the PATH. (/usr/bin/git)","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning theia websocket reconnecting on","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning on theia websocket first time took: 1590.8 ms","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"warn","time":"2020-10-20T13:28:39.059Z","message":"Frontend t.configure is slow, took: 2176.6 ms","payload":{"name":"root","params":[]}}

Additional information

Red Hat Linux 7.8 (3.10.0-1127.el7.x86_64)
Docker CE 19.03.13
Rancher Kubernetes 17.5.1 (Calico networking)
Gitpod v0.5.0

Example repository

n/a

@jgallucci32
Copy link
Contributor Author

It turns out it is a problem with the host networking configuration of the ws-sync pod. I had to add the following to its manifest as a workaround to get around this problem:

dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true

I am curious if other deployments see this or if this is specific to RKE due to its default pod security polies not allowing this.

@inventionlabsSydney
Copy link

Hi, I'm experiencing this exact issue @jgallucci32
Could you please explain further how you implemented your fix? I'm struggling here.

I am unsure where to apply

dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants