Transitioning DockerFile and Moby Build features

[mavenugo]

Based on the project discussions under
https://forums.mobyproject.org/t/topic-find-a-good-and-non-confusing-home-for-the-remaining-monolith/37/57

This issue is opened to track:

Moby Core aims to remove all dependencies on Docker products and Docker trademark. 
This is important for criteria 1: clarify the relationship to Docker. 
That includes removing terms like Dockerfile and docker-compose.yml from the Moby Core API 
and its specification.

This would have to be done gradually, and in a way that provides continuity for current API users. 
It would probably involve deprecating the current /build API endpoint (which badly needs an overhaul anyway) 
and either replacing it with something more flexible, or taking it out of scope entirely and 
leave it to the higher layers to implement build. 
This is the path that Kubernetes has chosen for example. There are several active projects 
implementing higher-level container build primitives decoupled from the underlying execution engine.

To be clear we should work on this in parallel to the project renaming. 
Consider this call for volunteers to propose a transition plan for Moby Core build features.

[mavenugo]

#32925 is tracking the technical details of Buildkit. We can use this issue to track the transition for Moby Core build feature out of moby-core and to the downstream projects.

[tonistiigi]

Some more background how the buildkit integration would work with Dockerfile split.

Buildkit design already splits build definition and builder invocation. It supports a concept of frontends that are components for preparing user-friendly formats to a general purpose low-level definition.

In a simplified description, this low-level definition is a graph of build steps with multiple inputs and outputs. Only fundamental actions are supported like sandboxed execution and access to remote sources. There isn't anything Dockerfile specific in there like ENV, ARG, USER etc. All the builder functions like execution scheduling, instruction cache, import/export of data and cache etc only run on the low-level definition, meaning any other format automatically gets the same benefits.

Buildkit development happens in a separate repository. When it has matured enough to have feature parity and stability it can be integrated with Moby core API(or more generally under the umbrella of Moby APIs). Dockerfile frontend would be put to some other repository and maintaining the Dockerfile format will continue to happen there. If Docker wants, they can also have a more higher level or opinionated API in their products. The integration of buildkit with the rest of the Moby APIs also depends on the containerd integration and Moby module splitting efforts, but buildkit itself should be flexible enough not to have specific requirements on how they need to be implemented.

/cc @AkihiroSuda

[andrewhsu]

I think this issue has run its course. I suggest we close this issue.

[AkihiroSuda]

@andrewhsu @tonistiigi

Will Dockerfile remain under moby/buildkit repo? Who are the maintainers of the Dockerfile specification?

[thaJeztah]

I think the plan was to move the spec to its own repository at some point, but that it was easier to put it in buildkit for the time being