supervisor 502 errors causes indefinite delays

[apolopena]

Describe the bug

This is an intermittant bug that occurs often with many types of repos and gitpod workspaces in a number of conditions.

• The workspace hangs indefintely either before, after or during the "Getting Docker Image 2/3" message but never during the "initializing workspace" message.
• Eventually the workspace IDE may open or it may not however the workspace has a status of RUNNING and minutes are charged against the users account even when they cannot access the workspace.
• The workspace in question can always be stopped or deleted from the dashboard.

Typical errors in the console look like the below image. In this particular case the workspace IDE finally did open but the delay was unacceptable, most users would give up after 12 failed attempts to load the supervisor and no status message. The Theia websocket took 2.474 minutes to open. All this time gets charged to the user.

Steps to reproduce

The bug is intermittant but frequent. I have not found reproducable steps yet. This issue seems to only mostly happen after a workspace image is cached and can happen when starting the workspace, building it or reloading it once its already started.

Expected behavior

The fetch using GET for /supervisor/v1/status/supervisor should not fail with a 502 Bad Gateway error. If there is a sizable delay as the system retries to get the supervisor. The user should be notified about this indefinite delay so they can more easily ask for assistance from gitpod support and the issue can be pinpointed much easier.

Additional information

I am on the US03 cluster. My gitpod account name is Devz3n

Example repository

The bug is intermittant and does not appear to be tied to a specific repo however the issue occurs frequently with this repo.

[csweichel]

Thank you for your detailed report.

I find the blobserve cache misses most troublesome. They definitely should not happen.
Will investigate.

[csweichel]

After investigating this further it looks like the workspace in question, and judging from your description, what you're experiencing is a too long image pull. This is an issue we're tackling at the moment (#2513).

The 502 errors you're seeing are expected. We have the frontend do an eager kind of readiness probe that fails until the workspace is running.

Note: you will not be charged until the workspace is actually running, i.e. the IDE is ready to be used. All the time you spend on the "loading screen" and is not being charged.

[apolopena]

@csweichel Good to know, thank you.

I have questions in regard to your comment about time charged since this bug often results when the workspace is running (according to the dashboard):

Note: you will not be charged until the workspace is actually running, i.e. the IDE is ready to be used. All the time you spend on the "loading screen" and is not being charged.

So does this means that usage time is only charged for when Theia is open in the browser and, not for when the workspace is running?

For example: I start up a workspace and I wait 7 minutes to get in and the IDE never opens. I then go to the dashboard and it says the workspace has been running for 7 minutes. Will I be charged for 7 minutes or until I stop that workspace that I cannot open? Or is it possible to be charged less time than what the dashboard says for a "Running" workspace?

[akosyakov]

So does this means that usage time is only charged for when Theia is open in the browser and, not for when the workspace is running?

You are getting charged when the workspace is actually running, but not for time to prepare it, like acquiring a node, pulling docker images and so on.

For example: I start up a workspace and I wait 7 minutes to get in and the IDE never opens. I then go to the dashboard and it says the workspace has been running for 7 minutes. Will I be charged for 7 minutes or until I stop that workspace that I cannot open? Or is it possible to be charged less time than what the dashboard says for a "Running" workspace?

Dashboard displays the total time not only running. You can see a break down of usage here: https://gitpod.io/usage/ Please look at it to find out how much Gitpod hours actually was consumed by a concrete workspace.

Please also have a look at these docs:

• https://www.gitpod.io/docs/subscriptions/
• https://www.gitpod.io/docs/life-of-workspace/#timeouts

[apolopena]

Thanks for the clarification. It can be difficult to equate actual usage vs overall running time.