Unexpected error loading prebuild

[aledbf]

Bug description

rpc error: code = FailedPrecondition desc = cannot initialize workspace: prebuild initializer: Git fallback: git initializer gitClone: mkdir /dst/spring-petclinic: no such file or directory

Workspace affected

gitpodio-springpetclini-8a38a5a57eu

Expected behavior

1. Log when this happens (we don't now).
2. Ideally we'd wait longer until the file is ready, before trying to clone

Example repository

None

Anything else?

How long do we wait before the file system is ready?

[sagor999]

We already have log entry for this:

gitpod/components/content-service/pkg/initializer/git.go

Line 75 in 6390f20

log.WithError(err).WithField("location", ws.Location).Error("cannot create directory")

Also looking at logs, it looks like workspace failed, and then 5 minutes later ws-daemon tried to run initializer for it (???):

I cannot quite understand what exactly happened to that workspace lifecycle.

[kylos101]

Thanks for looking at this one, @sagor999 ! @jenting prior to resuming PVC work, could you peek at this to see what you can find? It's the last of broken windows we found from gen59 traces.

[jenting]

1. Log when this happens (we didn't for now).

We log it already

2. Ideally we'd wait longer until the file is ready, before trying to clone

We haven't run git clone; the error reports on os.MkdirAll(ws.Location, 0775) failed 🤔

---

I thought the error happens at this line

gitpod/components/content-service/pkg/initializer/git.go

Line 75 in 3d97d55

log.WithError(err).WithField("location", ws.Location).Error("cannot create directory")

However, I did not find the span isGitWS in the tracing

gitpod/components/content-service/pkg/initializer/git.go

Line 64 in 3d97d55

span.SetTag("isGitWS", isGitWS)

🤔

[jenting]

I'm blocked for this issue now.

I can't reproduce it locally to make the os.MkdirAll(ws.Location, 0775) failed with the reason no such file or directory. Is this because the mount point /dst/ is not ready yet?