StopWorkspace is called prior to regular workspace transitioning to running state

[sagor999]

Bug description

StopWorkspace is called on workspace that has not finished initializing yet, causing ws-daemon to fail content init 10 min later.
Not sure why it fails 10 min after StopWorkspace was called, most probably some sort of timeout triggers somewhere.
Workspace logs:
https://cloudlogging.app.goo.gl/1qNcUN3hcr8GmPE16

Webapp logs:
https://cloudlogging.app.goo.gl/T344gi8ZHzDLLJNZ9

@geropl why does webapp is calling StopWorkspace prior to workspace starting up? I thought it was only implemented for prebuilds, but this happens on regular workspaces as well.

For workspace team:
we need to be able to handle this gracefully. This would be related to #11453 but that one specifically is looking to address prebuild cancellation, not regular workspaces. It is possible that work done in #11453 might address this issue as well, but most probably it will need additional work.

Steps to reproduce

not sure. need input from webapp team

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[sagor999]

This happens because there is no way to signal to ws-daemon to stop InitWorkspace process.

[utam0k]

I saw the same error.
I am wondering why there is an error about /dst so long after the request for a stop.

[kylos101]

@sagor999 how often is this happening? What is the impact to users if we don't fix?

For the meantime, I am going to assume infrequent, but add to the webapp inbox and mention Gero. This way, they can inspect the logs more leisurely using their flow.

If this happening often and negatively impacting users, let us know? I'll leave in our inbox for now to wait till I hear back.

[kylos101]

@geropl adding this to the webapp inbox. Can we ask for webapp help in determining potential steps to reproduce?

[geropl]

how often is this happening? What is the impact to users if we don't fix?

Can we ask for webapp help in determining potential steps to reproduce?

For prebuilds, this happens regularly now. Whenever a new commit is pushed, we cancel prior prebuilds on said branch. This should be reproduceable in a preview env with a script to push to a repo that is set up as project before.

Especially in cases with a lot of traffic (main branches + long builds) this happens nearly every time. For some teams this means that a lot of their prebuilds fail with weird error.

[kylos101]

@geropl thank you! That'll help us recreate, I think.

@sagor999 I am going to add this to breakdown for now, so we can socialize later this week.

[kylos101]

Removing from scheduled groundwork for now, not sure how often this is happening.

[geropl]

Related PR: #12386

@sagor999 I think this problem might be resolved by that, right? 🤔