Composition patterns for Moby tool

[deitch]

This is a general topic, opened per request by @justincormack, to discuss composition patterns. It also is intended as a general discussion place for the right way to make things capable like #89 opened by @samoht .

/cc: @rn

In docker, if we need to add some binaries (e.g. ssh client or git) to an existing standard image (e.g. nginx), we use a Dockerfile and extend it with FROM.

For building linuxkit images using moby, we may not want or be able to do so with custom extension images. Moby's purpose is a tool to "compose" to "weave" together various OCI images to create a new image. In the case of LinuxKit, that is a bootable OS image (although it could be generalized to other non-OS cases as well).

Moby currently supports the following composition patterns:

• Single kernel, init, onboot, services, files
• files as stacked (all appear on "root" filesystem, sequential)
• init as stacked (all appear on "root" filesystem, sequential)
• onboot and services as separate (each in own filesystem, parallel)

#89 was requesting a new pattern, wherein onboot and/or services can be stacked. Essentially, it is trying to solve the "I need this standard image with the capabilities of a few more mixed in." It is similar to the mixin capabilities of some programming languages.

Another example is tools availability on login. The standard linuxkit/getty and linuxkit/sshd images provide standard services. If someone needs, e.g. the git client or some other tools available at login, the only ways to provide them are messy:

• Install in the underlying host and bind-mount - brittle and pollutes the underlying host filesystem, undesired
• extend the getty or sshd image with your own which adds the tools, and use that in your .yml instead of standard getty or sshd - also brittle and easy to lose track, when all you want is one package

#89 proposed stacking layers for services like we stack for onboot. There are alternate ways of doing this. However, I believe we do need a way for moby to weave together images.

In a larger sense, I think moby needs to be able to weave without config files at all, e.g. moby build https://github.com/foo/bar/foo.yml --std getty,git --files ...

That may not be the best example, and there are ways to change that, but the general sense of weaving together existing "templates" and images without actually extending each one is important. This could be generalized to docker build ... as well?

[justincormack]

The assumption has been that there might well be another stage that covers the assembly of each package that happens before final assembly, that the tool would also control. That would include teh functionality of the Makefiles in pkg/ at least. Plus ideally more reproducibility support. Possibly driving a more scriptable underlying docker build process, although it could be done by generating Dockerfiles now I guess.

(How many stages it actually ends up as is less clear, probably more a UI issue; now the tool is being split out into a library that can be split out more)

[deitch]

That would include teh functionality of the Makefiles in pkg/ at least

I am not thinking about constructing , e.g. linuxkit/init or linuxkit/sshd, but rather the inclusion of it in a moby build ....

It probably isn't correct to refer to that is runtime, since runtime really is linuxkit run ... (or even launching an AWS instance). There are (at least) three stages here (feel free to debate number and nomenclature):

• build - when we create linuxkit/init or linuxkit/getty
• compose - when we run moby build ... to compose together a runnable linuxkit image
• run - when we do linuxkit run ...

I refer specifically to compose time, how we compose the linuxkit image.

Or are you suggesting that instead of the current flow of (a) we build linuxkit/foo packages and send them to the registry followed later by (b) end-user runs moby build and pulls them down; we would instead have a process where you just git clone <repo> and the whole thing gets built - OCI images and all leading to linuxkit image - at compose time? build time == compose time?

Separately, I agree wholeheartedly that the "build" process (using the above language) could use a lot in reproducibility and ease of construction.

[richardj-bsquare]

Just found this discussion after investigating LinuxKit to the point where I had a 'nfs-client' pkg and the docker-ce-dind pulled in...

Now I want to expose these two services (and their binaries) to the SSHD service, and find that this is not at all easy to do (in fact I can't work it out with the current tooling, perhaps someone can enlighten me as to how it can even be achieved, it probably has something to do with bind mounts, but I can't work it out from the current documentation, an example/use-case perhaps).

The solution (suggested in #89) of producing a docker image that does all 3 things seems to go completely against the 'do one thing and do it well' approach that has been the mainstay of docker images so far. Starting docker and SSH and NFS within a single docker image means introducing extra levels of unnecessary complexity.

It seems completely sensible to be able to write an ssh service that borrows what it needs from the other services, i.e. a mixin like approach.

[justincormack]

We should move this to LinuxKit repo.