Roadmap for `docker deploy` and Bundles

Bundles

Bundles are a multi-image redistributable format that describe multi-container distributed applications (or stacks) running on Docker. A bundle is akin to a “multi-image image,” and deploying it gives rise to a “stack” consisting of several communicating microservices. Conceptually, this is analogous to Docker images today:

	Human-editable Description	Binary Artifact	Running Instantiation
Bundle	docker-compose.yml	Bundle	Stack
Image	Dockerfile	Image	Container

In June 2016, Docker announced Distributed Application Bundle (DAB) files, a predecessor to bundles, as part of Docker 1.12. DAB is the file format used internally to represent bundles.

This issue outlines the roadmap for bundles, the file formats involved, how they relate to one another, how bundles relate to existing multi-container tools like Docker Compose and how the CLI/API will be affected.

Docker 1.12 Concepts: Swarm Mode, Bundles and docker deploy

Engine swarm mode orchestration (Docker services). Since Docker 1.12, Engine has supported built-in container orchestration through the addition of Docker Services (docker service and related subcommands). A Service in Docker is a replicated set of 1 or more identical containers. Loosely speaking, a Service in Docker models the concept of a “microservice” within a larger application.

Bundles. A bundle can be generated from a Compose v2 file. The bundle itself is a redistributable binary format, and in the future docker-compose will be just one possible build tool. When a bundle is deployed to a cluster, its running instantiation is called a stack.

docker deploy. This subcommand is used in 1.12 to deploy a Distributed Application Bundle (DAB, or simply bundle) file.

Roadmap: Compose files, Bundle files, Daemon-side Bundles and docker stacks

As of Docker 1.12, docker-compose is currently the tool used to create bundle files. A bundle file is generated from a Compose v2 file using docker-compose bundle [-f docker-compose.yml]. It is then deployed using the docker deploy subcommand: docker deploy --file mystack.dab. A more complete description of this 1.12 functionality is available here.

In Docker 1.13/1.14, our intention is to shift away from using docker-compose to generate bundles. Instead, the docker CLI itself will generate bundles from Compose v2 input files using a new docker bundle subcommand. Example:

docker bundle build [-f COMPOSE_FILE] -t BUNDLE

In addition, Docker will move toward a system in which bundles are stored and managed internally by the Engine rather than as user-managed .dab files. For instance, the user will be able to push and pull bundles:

docker bundle push BUNDLE                   (see below)
docker bundle pull BUNDLE                   (see below)

Other docker bundle commands like bundle ls, bundle tag and bundle rm will allow the user to manage these daemon-side bundles.

Deployment and management of running stacks will be done using the top level docker stack subcommand (which already existed in 1.12 and will be extended in 1.13/1.14). Deployment of a bundle to instantiate a stack will be done using docker stack deploy; docker deploy will be aliased to this command for backward compatibility. For management, existing commands such as docker stack ps, docker stack inspect and docker stack rm will be available to view and manipulate running stacks.

Example Workflows

Running a stack from a Compose file

docker bundle build [-f COMPOSE_FILE] -t BUNDLE
docker stack deploy [-c STACK_CONFIG] BUNDLE STACK

Running a stack from a registry

docker stack deploy [-c STACK_CONFIG] BUNDLE STACK

[thaJeztah]

Thanks @mgoelzer. Perhaps we should add this to the ROADMAP.md (I think that was mentioned elsewhere); https://github.com/docker/docker/blob/master/ROADMAP.md. wdyt?

@thaJeztah Yes, I want to update ROADMAP.md more broadly. I think we should aim to move this issue's content into an updated ROADMAP.md and then close out this issue in favor of ROADMAP.md

[nathanleclaire]

Thanks Mike for the detailed summary. Got a couple questions:

1. So I understand that one of the goals of DAB is to prevent environment-specific configuration from leaking into descriptions of "stacks", but inevitably you will have to set this somehow -- how?

2. DAB is attempting to solve portability problems but it is relying on a remote registry to do so. Relying on a remote registry is itself something which cannot be safely done portably. Each network has its own configuration and idiosyncrasies. Corporate firewall / proxy / VPN and more is common and will break DAB. Is "it's portable if your network is configured wide open" really portable?

3. Have you considered that mixing in Compose V2 to all of this might end up causing more trouble than it solves? I know it's tempting ("there are so many existing Compose files"), but how many of those existing applications can actually be translated into bundle files without a ton of work? (And if you have to re-work your Compose YAML significantly to shoehorn it into DAB, what are we gaining here vs. creating a new spec altogether?)

   Consider a beginning user's journey with Compose. It almost always starts and ends with environment-specific config and, in my experience, this extremely rapidly becomes habit for users writing Compose files. If you want to take this very simple Compose file and turn it into a DAB you run into problems almost immediately.

   1. build key doesn't work, only image. OK, I guess these 32,000 Docker Compose files cannot be translated to DAB without modification? So, at least roughly 50% of existing Docker Compose files on GitHub will not translate to DAB without modification.
   2. volumes are ignored (and ~39,000 / 60000 total Compose files on GitHub seem to reference volumes!). So what do we win by using Compose for this? Huge swaths of Compose files will not work out of the box.
   3. Looks like in the generated DAB a network is added in? How exactly is that portable? It's not safe to assume that docker network setup looks the same everywhere... There might not be a bridge, there might be overlays configured a particular way by an administrator, there might be specific plugins the user wants, and so on.

4. Why not just have a Compose V3 file which maps directly to docker service flags and related mounts, etc.? That'd be very easy to reason about (almost direct translation) and it's the simplest possible thing, so why not? Personally I'm chomping at the bit to just get a YAML file which maps almost directly to docker service create/update. The current proposal, instead of focusing on simplicity, adds at least 2 or 3 more steps into the process. (Either this, or a new "schema file" defining the types and arguments for a stack create and a file per-environment to define those arguments).

5. As for the content addressability aspect, shouldn't operators who want that run images with Docker content trust enabled? Why force content addressability in the deployment spec layer? Without actual cryptographic identity verification, what does having these image content hashes in a JSON file win us? Wouldn't mucking with a plaintext JSON file be comparatively pretty easy for a motivated attacker?

The current DAB method does have some interesting properties, however I have a hard time imagining myself selling this to a potential user: "DAB is great! To use it, just write a Compose file, and then immediately translate that Compose file to a new format, since the Compose file format is not supported. Oh, you have one already? Nice, it'll still need to be modified heavily to be translatable to DAB ... Oh, and make sure your images are pushed first. That's important, gotta have the images pushed, can't experiment with this technology locally at all. And also you are going to have to define volumes and networks somehow after the fact anyway."

Ping @dnephin

[dnephin]

So I understand that one of the goals of DAB is to prevent environment-specific configuration from leaking into descriptions of "stacks", but inevitably you will have to set this somehow -- how?

Currently by doing docker service update, which I don't think is a good way of doing it. I personally think we need a second format for including all the non-portable stuff. This second "settings" file wouldn't be distributed from a registry, but would be managed by the local operator. We could have tooling to create a default "settings" file based on hints provided from a bundle. #26748 is an issue for this.

Is "it's portable if your network is configured wide open" really portable?

This is fixed by our work to move bundles to be an "internally managed" object of the engine. The image layers referenced from a bundle must be on the same registry as the bundle manifest, so if you can fetch the bundle, you can fetch all the layers required to run the services in the bundle.

Have you considered that mixing in Compose V2 to all of this might end up causing more trouble than it solves?

Yes. I agree with many of your points. I personally think that eventually we need a more appropriate "source" format, but right now that format is Compose v2, and creating 3 formats all at once seems like a bad idea. For now we're focusing on the bundle format. Once that's stabilised we can look at improving the source format , and making it more appropriate for translation into a bundle.

what are we gaining here vs. creating a new spec altogether?

Time to get the source format right. If we build it all at once there are too many things in motion for anyone to make use of it.

Looks like in the generated DAB a network is added in? How exactly is that portable?

It just uses the default network driver configured by the engine. I believe all networks should provide the same things to containers.

Why not just have a Compose V3 file which maps directly to docker service flags and related mounts, etc.?

I wouldn't be surprised if that's what we end up with. Right now Compose does not work with swarm mode, so it wouldn't make sense to create a format that doesn't work with the tool that reads it.

As for the content addressability aspect,

You lost me with this one, but I think the idea is that the bundle itself will be signed, you don't have to worry about signing the individual images within it.

The current DAB method does have some interesting properties, however I have a hard time imagining myself selling this to a potential user

Yes, that's why they are experimental. They still need work to be more useful. Some of that work will be done for 1.13. Most of that work needs to be completed before it can leave experimental.

[nathanleclaire]

I wouldn't be surprised if that's what we end up with. Right now Compose does not work with swarm mode, so it wouldn't make sense to create a format that doesn't work with the tool that reads it.

To be clear, I wasn't suggesting that we have Compose read in the "V3" files, but rather a new YAML format similar to the existing stuff (but aimed directly at docker service) that docker CLI itself reads in and translates directly to service create and/or update.

You lost me with this one, but I think the idea is that the bundle itself will be signed, you don't have to worry about signing the individual images within it.

Why would DAB would take this responsibility though? Why would a tool for orchestrating containers/tasks concern itself with image provenance? And what exactly is a bundle in this context? The images themselves ? The DAB file?

It just uses the default network driver configured by the engine. I believe all networks should provide the same things to containers.

What about things like network IPAM config (https://docs.docker.com/compose/compose-file/#/ipam)? And plugins? If networks are bundled that (plugin) will be yet another dependency that the bundle will need present in order to be executable right?

@nathanleclaire

1, 3 and 4) I agree with you. I too think we need a Compose-like YAML format for docker services. However, bundles are trying to solve a somewhat different problem, which is how do you define a multi-tier application in an abstract enough way that it is consistently portable across environments?

2. But don't you think the concept of "portability" implies that you've successfully obtained the bundle already? It will always be the case that if you cannot obtain some artifact (dab, jar, whatever) then you cannot run it in that environment.

3. Good point. The hash here should never be used for cryptographic verification because it's not a signature. The hash ensures reproducibility of the bundle -- e.g., redis:latest may be different in different environments but redis@sha256:1234567890abcdef is unambiguous in all environments. But for DCT, you'd want signatures on both the bundle manifest and the images so that you can revoke an image's signature independently. In the revocation case, the bundle as a whole should not run, even though the sha256 hash will still be correct.

3-iii) I agree the abstraction sometimes gets leaky around networks, especially for enterprise configurations. Ideally, a container network is just some software-defined abstraction that can be realized on top of any properly configured environment. In practice, this is not the case because some network drivers want to do things like make container IP allocation decisions based on what physical node they are running on. This is a challenge for portability in general.

[nathanleclaire]

1, 3 and 4) I agree with you. I too think we need a Compose-like YAML format for docker services. However, bundles are trying to solve a somewhat different problem, which is how do you define a multi-tier application in an abstract enough way that it is consistently portable across environments?

I think that maybe where it needs to head is that the "bundle" file defines a "schema" which the running end needs to "invoke" with the correct parameters. Dan I think kind of hints at this in one of his comments, and it would fit in with the Glyph article I linked's line of thinking ("A Container is a Function Call").

I just haven't seen anything in the current bundle spec indicating things like "this container must have a volume mounted in at /foo". Not saying it's not planned, just don't see any of that kind of detail today so far. And sketching out blueprints how these kinds of things might work or look like always unearths nasty little corner cases that weren't considered in the original vision.

But don't you think the concept of "portability" implies that you've successfully obtained the bundle already? It will always be the case that if you cannot obtain some artifact (dab, jar, whatever) then you cannot run it in that environment.

So will users be able to obtain just a bundle artifact and execute it independent of distributing the bundles and/or their images via registry? i.e., will there be a registry-free workflow possible like there is for Docker images (docker save/docker load). Because the bundle idea seems inherent coupled to Engine internals and to the registry today.

The hash ensures reproducibility of the bundle -- e.g., redis:latest may be different in different environments but redis@sha256:1234567890abcdef is unambiguous in all environments

Maybe a little nitpicky, but wouldn't this be better solved by immutable image tags? Sure, redis@sha256:1234567890abcdef might tell me as an operator that I'm unable to obtain the exact image I'm intended to run. But how does that help me? Think about git tags, for instance. Which is more useful: Me handing you e18a9193d49de4a45ba4b1b6379cad7775481cd2 and telling you that's the docker/machine revision you want, or me telling you that you want the v0.8.2 tag?

In practice, this is not the case because some network drivers want to do things like make container IP allocation decisions based on what physical node they are running on. This is a challenge for portability in general.

Right, so how will bundle solve it?

Thanks for your commentary and willingness to answer questions @dnephin @mgoelzer !!