Config update for buildkit

[alexellis]

Expected Behaviour

It appears that buildkit should be able to run on Kubernetes using both non-root and unprivileged container security. A configuration should be put together for this as soon as possible.

Possible Solution

Update the files in ./yaml/ for the builder pod.

Steps to Reproduce (to test)

1. Update the config/container
2. Trigger a build with all of the templates available in the Community Cloud Cluster
3. See if it worked

Context

Increasing security.

Impact to users

Recommendation to upgrade, a rolling update should be possible.

[alexellis]

@AkihiroSuda @matipan

[matipan]

Derek assign: me

[alexellis]

Thank you

[matipan]

I've been trying to get the rootless buildkit working on GKE but it seems that Buildkit does not work if you use the Container Optimized OS for the nodes. See this issue. According to a comment from @AkihiroSuda it does work when using Ubuntu nodes and the buildkit:v0.4.0-rootless image but I have not tried it yet.

Should we still do the upgrade for using the rootless mode and warn GKE users to use Ubuntu or should we wait until rootless bulidkit is more stable? According to this doc it's still experimental

[alexellis]

I think your suggestion sounds good. In any case the builder could be run in a separate node pool with an Ubuntu machine.

[alexellis]

Sounds like we have a way forward?