Skip to content

cannot initialize workspace: cannot connect to ws-sync #2029

Closed

Description

@jgallucci32

Describe the bug

Gitpod cannot initial workspace with the following error

cannot initialize workspace: cannot connect to ws-sync: cannot connect to ws-sync: cannot connect to workspace sync

Further investigation of the logs of the deployed workspace show the Theia server immediately shutting down

{"component":"workspace","severity":"info","time":"2020-10-19T19:59:54.249Z","message":"Registered built-in plugin","payload":{"name":"root","params":[{"name":"vscode.docker@1.47.3","loc":"/theia/node_modules/@gitpod/gitpod-ide/plugins/vscode.docker-1.47.3.vsix"}]}} 
{"message":"Theia supervisor loop ended - waiting for Theia to come down","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"} 
{"message":"shutting down health endpoint","serviceContext":{"service":"theia-supervisor","version":""},"severity":"info","time":"2020-10-19T19:59:58Z"} 
{"error":null,"message":"Theia was stopped","serviceContext":{"service":"theia-supervisor","version":""},"severity":"warning","time":"2020-10-19T19:59:58Z"} 

What is interesting is this error does not occur 100% of the time. Applying changes to the helm chart causing certain pods to redeploy will allow it to work. It does not occur when using a single-node K8s cluster, so perhaps we have a scheduling issue where services are not talking across nodes correctly.

Steps to reproduce

  1. Deploy Gitpod to 2 or more Kubernetes nodes
  2. Apply changes to helm chart (sometimes this breaks it, sometimes this fixes it)
  3. If error exists, keep applying changes and/or redeploy one or more pods until services restore

Expected behavior

Workspace should initialize and the following should appear in the logs rather than the shutdown of Theia

{"component":"workspace","severity":"info","time":"2020-10-20T13:28:38.586Z","message":"Using Git [2.26.0] from the PATH. (/usr/bin/git)","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning theia websocket reconnecting on","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"info","time":"2020-10-20T13:28:39.057Z","message":"Turning on theia websocket first time took: 1590.8 ms","payload":{"name":"root","params":[]}}
{"component":"workspace","severity":"warn","time":"2020-10-20T13:28:39.059Z","message":"Frontend t.configure is slow, took: 2176.6 ms","payload":{"name":"root","params":[]}}

Additional information

Red Hat Linux 7.8 (3.10.0-1127.el7.x86_64)
Docker CE 19.03.13
Rancher Kubernetes 17.5.1 (Calico networking)
Gitpod v0.5.0

Example repository

n/a

Activity

  1. jgallucci32 commented on Oct 20, 2020

    @jgallucci32
    ContributorAuthor

    It turns out it is a problem with the host networking configuration of the ws-sync pod. I had to add the following to its manifest as a workaround to get around this problem:

    dnsPolicy: ClusterFirstWithHostNet
    hostNetwork: true
    

    I am curious if other deployments see this or if this is specific to RKE due to its default pod security polies not allowing this.

  2. inventionlabsSydney commented on Apr 5, 2021

    @inventionlabsSydney

    Hi, I'm experiencing this exact issue @jgallucci32
    Could you please explain further how you implemented your fix? I'm struggling here.

    I am unsure where to apply

    dnsPolicy: ClusterFirstWithHostNet
    hostNetwork: true
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions