buildx library as a higher-level buildkit client

[errordeveloper]

Problem statement

When I looked at using buildkit API from a Go program, I ended up comparing what buildctl does to what buildx does and very quickly realised that buildx is a lot more advanced. I find it hard to tell what pieces I may need to borrow to create a lightweight and robust client that is able to do nearly as much as buildx.

However, currently it's not very trivial to just call buidx functions, one needs to figure out how to setup the drivers the right way, and may need to also use github.com/docker/cli/cli/command and github.com/docker/buildx/store, which do seem like very desirable dependecies for just any applications that might need embedding a buildkit client. Namely, command.Cli appeare very Docker-spefic, and store package is very specific to buildx's state management needs as CLI.

In my case I have a set of addreesses of remote buildkit deamons and TLS credentials (as structs like struct { ca, cert, key []byte }), and I'd like to either setup generic gRPC clients for each of these and create a buildkit client from that, or otherwise at very least setup buildkit client without owning the gRPC client setup also.

For this case I find that buildx remote driver abstraction is not very useful, and it also took me a lot of time to figure out how to set it up. However, I do find that build.Options struct and most of the logic in build.BuildWithResultHandler is quite useful, especially toSolveOpt appears to have what most comprehensive clients need to do.

I certainly don't want to copy some code from buildx and then maintain it. Neither I want to shell-out (for multiple reasons).

Proposal

Povide a simpler API that does most things that build package currently offers, but without CLI-centric assumptions and design aroind an equivalent of remote driver (i.e. initialised with a gRPC client or a URL). This API doesn't have to offer strong stability guarantees, it could be evolving in the way that buildx project currently does, but it's best if it stays disentanlged from the CLI use-case.

type BuildClient stuct {
  	Platforms     []specs.Platform
        Conn grpc.ClientConn
        Options Options
}

type BuildClientSet []BuildClient

func (*BuildClientSet) BuildWithResultHandler(ctx context.Context, w progress.Writer) (BuildResult, error)
func (*BuildClientSet) BuildWithResultHandler(ctx context.Context, w progress.Writer, resultHandleFunc resultHandler) BuildResult, error)

type resultHandler func(*BuildClient)

type BuildResult struct {
}

I am not sure what BuildResult should look like, but ideally it shouldn't be a nested map like it is right now, it needs to be more structured.

[tonistiigi]

Buildx first and foremost, is a CLI tool. There is a Go client library in BuildKit and I don't think it makes sense for us to maintain two different sets of Go libraries that do the same thing. The amount of Go devs who want to write their own code to interact with BuildKit instead of calling Docker tools directly isn't so big that a different library needs to exist for every taste or level of abstraction(or other people can do it, for example Dagger who have written their own proxy variant). If the BuildKit client lib can be improved, we should do the work there. This shouldn't compromise on the capabilities that the current library has, just to make it simpler.

Buildx does separate build and CLI flag parsing into separate packages for keeping the code cleaner and because build pkg is used by multiple CLI commands. This does not mean, though, that build pkg is built to be reusable outside of Buildx itself or that any effort goes into maintaining the backward compatibility of that pkg.

Adding some very specific extra abstractions like the gRPC struct doesn't probably improve the general reusability but may increase the maintenance cost for future buildx dev. The most important goal of this repo is to allow us to iterate quickly on opinionated user-facing build features.

--

If your problem is that some of the existing drivers are not powerful enough, we should really extend the capabilities of that driver, not try to reimplement buildx functionality in another tool. Every time build pkg is imported directly it increases fragmentation for the Docker users as that pkg defines the expected "docker ux" and if it doesn't match with the version of the Docker installation the user has installed, it will have unexpected results.

[errordeveloper]

If your problem is that some of the existing drivers are not powerful enough...

No, actually very much the opposite, I just find that the driver abstraction is not useful for deploying builders, the remote driver is already a good example, the interface is very foced there. Stateful parts are also unhelpful in my use-case, I need to be able to create multiple clients from a single long-running process, and I would rather not worry about state that these clients may write to disk and accidentily collide with one another.

Adding some very specific extra abstractions like the gRPC struct...

My suggestion to expose grpc.ClientConn was based on my understanding on idiomatic Go. Right now gRPC client is hidden in dirver code, and passing any options is just impossible, in particular I need to be able to specify TLS credentials as []byte instead of files on disk.

... doesn't probably improve the general reusability but may increase the maintenance cost for future buildx dev.

I cannot possibly agree with this. The current shape of build package is huge burden, the refactoring I am offering to do was aimed at improving that code in the first place.

There is a Go client library in BuildKit

I'd use it, by why is invoking it takes so many lines of code in buildx?

BuildKit client lib can be improved, we should do the work there.

I don't mind the location, just seemed like buildx had the code already that should be easy enough to tease apart from some assumptions.

[tonistiigi]

I'd use it, by why is invoking it takes so many lines of code in buildx?

Because buildx implements many more features that are out of scope for buildkit, like drivers (and driver-specific exceptions in build), multi-targets, multi-node split & join, frontend fallbacks, old CLI compatibility etc. Buildx code is opinionated while BuildKit is general-purpose and reusable.

My suggestion to expose grpc.ClientConn was based on my understanding on idiomatic Go. Right now gRPC client is hidden in dirver code, and passing any options is just impossible, in particular I need to be able to specify TLS credentials as []byte instead of files on disk.

In driver interface, the Driver provides the whole client.Client instance that is already initialized with a specific clientconn. The configuration of it is left completely to the driver implementation.

What I was referring with my earlier comment was that there are a lot of possible low-level options that buildx does not expose because it doesn't use them. Adding one specific one would only solve one exception.

For the ClientConn specifically, the problem is that for buildx itself this is part of Driver responsibilities so any new options in build for that would be not used by buildx itself. If you change ClientConn to client.Client then maybe more public functions that take client.Client directly would be useful but it needs to be proven in build pkg as explained in #1466 (comment) . Also, #1377 could be a real issue as it requires configuring build requests where some requests go to different grpc clients(shared with other build targets) than others.

[errordeveloper]

Because buildx implements many more features that are out of scope for buildkit, like drivers (and driver-specific exceptions in build), multi-targets, multi-node split & join, frontend fallbacks, old CLI compatibility etc. Buildx code is opinionated while BuildKit is general-purpose and reusable.

I do understand this, and it's clear from buildx code that it was written to be a CLI. However, BuildKit client relies on very weak interfaces. It uses maps that appear to be documented only by examples, and most of the examples I've seen are so long, that I wouldn't dare to adopt. If the API was complex, but didn't rely on maps, I would be less hesitant to create my own abstactions. The arbitrary nature of maps is very daunting to build any abstractions upon. On the contrary, buildx has Options struct that provides basis that is much easier to reason about.
I don't personally mind where client abstraction is implemented, it just needs to use structs and the invovation should be a matter of creating a client for grpc.ClientConn, an calling build method with an options struct.