Expose more buildkit functionality through engine API. #38165
Description
I'd like to propose eventually adding a new Engine REST API which exposes buildkit's Solve more directly than /build which is focused around the dockerfile frontend specific use case and has some quirks due to that (and for legacy/historical reasons).
I've prototyped running a custom frontend via the /build API but it's a bit hacky and not terribly satisfactory (injecting a stub Dockerfile with a syntax directive in it, passing options by mapping to various Dockerfile-ish options etc).
Other potentially interesting things to expose are local exporter, debugger, shared session, multiple local sources and the ability to run a frontend client side (cf Build rather than Solve in the buildkit control API).
It's possible we could just extend on the current /build API to cover more underlying functionality (e.g. add a Frontend field to it), enhance the format of the returned body to remove the legacy framing etc.
Or we could perhaps expose a new endpoint mapping the underlying Solve call a little more directly that the /build endpoint, with most of the options passed through or lightly adjusted/filtered (e.g maybe the full set of Exporter/ExporterAttrs flexibility shouldn't be exposed?). The Session side would be the same as with /build (an upgraded conn to /session). The body returned would be the Status stream, I guess as a direct JSON stream of StatusResponse rather than wrapping in a JSONMessage like /build had to do (for compat reasons I suppose). I'm not entirely sure about SolveResponse, so I guess maybe there needs to be a little more structure to the body stream, but ideally not JSONMessage, just a simple union type map.
A possibly simple (but IMHO not at all ideal) would be a simple API end point which opens the entire control API gRPC via a /session-like endpoint. I don't think that is really what we want though (control API is not considered stable, might be too much low level power being exposed).