Expose the entire Build API through REST and introspection

[thedrow]

Currently it is only possible to build a docker image from a tarball containing a Dockerfile using the REST API.
There's no reason why we shouldn't be able to build a docker image entirely using the REST API.
This will enable integrations with multiple configuration management tools like Salt and Puppet which already have superior and less error prone provisioning capeabilities.
How hard would it be to achieve this?
It would involve exposing all commands in https://github.com/docker/docker/blob/master/builder/dispatchers.go if I understand correctly.

[tiborvass]

IMHO what would be even better, is to expose the build api to containers via daemon introspection. That way you could write shell scripts inside containers and salt and puppet would just have to call dockerfile commands. It's a different (and more difficult?) approach but I find it more powerful, since it would finally enable docker build to be a full-fledged docker run, and the Dockerfile/Dockerscript to behave as anyone would expect. WDYT?

[thedrow]

That will also work but it strikes me as more complex.
Why is it more powerful?
How would you provision a container with docker-py using that method for example? Eventually Salt will use it to provision containers. Puppet and Chef will use the ruby client and so on.
The REST API provides a convenient way to communicate with docker using any client or just an HTTP client (e.g. cURL).
How would the remote API look like with the method you proposed?

[thedrow]

@tiborvass ping?

[tiborvass]

@thedrow Don't get me wrong, I'm not necessarily opposed to an API for build as you described, I just wanted to point to daemon introspection as a possible solution.

I agree it seems a bit complex, but for me the major benefit is that it would be "jailbreaking" the buildtime: right now certain things are allowed in docker run (volumes, editing /etc/hosts, etc.) that are not possible with docker build. But when you think about it, building is nothing more than just running a container as specified by a Dockerfile and a context and saving the resulting image. Except, that you don't have all the expressiveness of docker run.

An equivalent solution today would be to do docker run --rm -v /bind:/mount myimage-builder | docker build -t myimage - where myimage-builder would stream the tar context to stdout.

If there were daemon introspection, you could imagine a sortof docker commit inside the container that would build the layers one after the other. Be it manual, or (more useful) scripted. There are a lot of things that need to be discussed (UI for introspection etc.) but how I currently imagine building a container: docker build -v /bind:/mount -t myimage where the Dockerfile could be some sort of Docker script or maybe even ANY script (shell, python, ruby, ...) that simply calls dockerfile commands. For instance:

$ dockerscript from ubuntu
$ dockerscript run echo hello from dockerscript

Keep in mind that it's far from clear for me what would be the best UI.

3rd parties could simply call a docker build API similar to the one we have today, and put their customizations in the Dockerscript (that would have a shebang):

$ cat Dockerscript
#!/bin/sh
dockerscript from ubuntu
text="hello from dockerscript"
dockerscript run echo $text

# there could be optimizations like accepting multiple commands across multiple lines:
dockerscript <<EOF
run echo foo
run echo bar
EOF

Sorry if you feel I'm hijacking your issue. We can talk more on IRC (handle: tibor).

[thedrow]

Any updates on this one?
I know that we have docker exec but that only helps us not having to run the provisioner inside the container which is not enough. I'd like to fully utilize the build caching mechanism that docker provides us with.

[tiborvass]

I'm closing this since #13171 was merged which was the last blocker for implementing #14298.

@jlhawn implemented an experimental client-side builder: https://github.com/jlhawn/dockramp which uses the new APIs in #13171.

Hope this helps :)