[image-builder] Get rid of the Gitpod layer

[shaal]

Bug description

This is something I see randomly, and I don't know where can I find any additional information about it.

Sometimes, when I open an existing gitpod-supported repo, I might see this screen below (Building Image)

I'm surprised to see it because I did not make any recent change in .gitpod.yml or in the custom image of the project.

Steps to reproduce

This is unexpected, I don't have steps to reproduce.
Perhaps it gets trigger after the official workspace-full gets updated, or after Gitpod update something else in the system?

Expected behavior

I wish that image build won't be a surprise.

If an image build process has to run and cannot be delayed, I'll be happy if it could run as soon as that update in the system was applied, before I open the workspace in my browser (similar to how prebuild work).

Example repository

No response

Anything else?

No response

[csweichel]

Indeed the current mode is not ideal. Those unexpected image builds happen whenever

1. the underlying image changes, e.g. with a .gitpod.yml like

image: gitpod/workspace-full:latest

2. the gitpod layer changes which is part of a system update.

This is a good next step after image-builder-mk3 is merged (#4812).

The best move would be to get rid of the gitpod layer altogether. With the advent of root in the workspace, we might have a path where we no longer need to defer privileged operations to the image build (e.g. creating the Gitpod user).

Caveat 1: one benefit of the current Gitpod layer implementation is that it transfers the workspace image to a registry under our control, which helps with reliability and workspace startup time. If we removed the Gitpod layer, we'd need to ensure we maintain this behaviour if we're not building the base image (example 1 above).

Caveat 2: while the Gitpod layer can go away, we'll still have other post-processing steps in the future (estargz). In those cases we could have image-builder return the unprocessed image and trigger the processing behind the scenes. Then the next time the image is needed, we'd have the optimised version available.

[csweichel]

/schedule

[csweichel]

/assign