feat(buildkit): Expose envd as a buildkit frontend

[gaocegege]

Mocker uses github.com/moby/buildkit/gateway/client instead of github.com/moby/buildkit/client.

It introduces a new abstraction Build func https://github.com/r2d4/mockerfile/blob/140c6a912bbfdae220febe59ab535ef0acba0e1f/pkg/build/build.go#L37

The benefit is that you can do some post-solve logic like this https://github.com/r2d4/mockerfile/blob/140c6a912bbfdae220febe59ab535ef0acba0e1f/pkg/build/build.go#L65

[gaocegege]

moby/buildkit#2916 (comment)

[gaocegege]

If we move to the frontend design, we can integrate with buildkit ecosystem seamlessly. For example, we can use buildx action in GitHub:

name: Name
on:
  push:
    branches: [dev]
jobs:
  push_to_registry:
    name: Push Docker image to GitHub Packages
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Push to GitHub Packages
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.CR_PAT }}
      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v2.0.1
        with:
          context: ./app/
          push: true
          file: ./app/build.envd
          build-args: |
             ...

[gaocegege]

Ref #510

[gaocegege]

Buildkit frontend requires a image from us, just like docker/dockerfile.

The image docker/dockerfile contains a CLI binary:

package main

func main() {
	if err := grpcclient.RunFromEnvironment(appcontext.Context(), dockerfile.Build); err != nil {
		logrus.Errorf("fatal error: %+v", err)
		panic(err)
	}
}

It wraps the dockerfile.Build with grpcclient.RunFromEnvironment. Function dockerfile.Build is the build func for Dockerfile.

We can provide a similar implementation in tensorchord/envd-frontend image if we make envd a frontend of buildkit.

package main

func main() {
	if err := grpcclient.RunFromEnvironment(appcontext.Context(), envd.Build); err != nil {
		logrus.Errorf("fatal error: %+v", err)
		panic(err)
	}
}

And the build.envd can be:

syntax=tensorchord/envd-frontend:<version>

We can use buildctl to build envd images with the help of the frontend:

buildctl build \
    --frontend gateway.v0 \
    --opt source=build.envd \
    --opt context=.

The gateway in buildkit will forward requests to our own tensorchord/envd-frontend, and transform frontend starlark statements to LLB.

[gaocegege]

After #606 is merged, we already have the build func. But we cannot make envd a frontend of buildkit:

• We rely on LocalDirs in SolveOpt to expose cache dir to the build process. The LocalDirs is not supported in build func. We should make cache dir a llb.Local, then mount it into the llb states. See https://github.com/r2d4/mockerfile/blob/master/pkg/build/build.go#L83
• We rely on containerimage.config and name in SolveOpt. SolveOpt is not exposed in the gateway client. We should use ResolveImageConfig in the gateway client to re-implement the logic.

[knight42]

We rely on LocalDirs in SolveOpt to expose cache dir to the build process. The LocalDirs is not supported in build func. We should make cache dir a llb.Local, then mount it into the llb states.

It seems we could leverage named context https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/ if we made envd a buildkit frontend, but we need to pass an extra argument to docker buildx build, like:

docker buildx build --build-context local-cache=~/.cache/envd ...

Ref: https://github.com/moby/buildkit/blob/23ab2d061932d3a14c5d6c84a2cf03558d442b4f/frontend/dockerfile/builder/build.go#L928-L967

[gaocegege]

The cache dir should be added by default in envd binary, Thus I think we could just make it in build func instead of a extra arg. 🤔

[knight42]

Please correct me if I missing something 🤔, we cannot access dir outside the build context dir during build time if envd is used as a buildkit frontend since buildx only adds the context dirs to LocalDirs https://github.com/docker/buildx/blob/701c548e46348da2958104907c9572ea7ce6ab52/build/build.go#L1376

[gaocegege]

Yep, we cannot access dirs if it is not added into buildkit as a build context.

I mean we should keep localdir in LocalDirs by default in envd's binary. If users use buildctl, they can use --build-context to add cache dir as build context.

[gaocegege]

BTW, if you are interested in buildkit integration with envd, maybe you can have a look at #124

We are working on supporting debug functionality in envd, similar to docker buildx. But there are many challenges especially from buildkit.