Epic: Distributable Compose apps

[aanand]

(This is closely related to #1784.)

The Docker image format, along with the Hub, registry and Dockerfile build system, constitute a really good way to distribute single-container software. There's no equivalent for multi-container applications. The closest you can get is a Compose file that references a bunch of images hosted elsewhere.

If there were a way to package up a whole multi-container app - its container's images and configuration - for one-step distribution and deployment (as simple as docker push, docker pull and docker run make distributing single-container software), it would enable much smoother distribution of much more complex apps than a Docker registry can currently supply you with.

This might entail designing an artifact that's generated from a Compose file, much as a container image is generated from a Dockerfile.

There are also interesting use cases regarding deployment and rollback - if you could serialise/deserialise between the state of an app and an immutable artifact, rolling back a deployment would become a single logical step.

[dnephin]

I've been thinking about this a bit, as it ties into #318. If the entire app can be pulled in that makes "including" it in another app much easier.

I was thinking along the lines of keeping the compose file and the images separate. That way you can re-use the docker-registry to pull the images, all you need is a way to distribute the docker-compose.yml. Keeping the images separate means you can pull things concurrently (instead of trying to pull a single large blob).

Sharing the docker-compose.yml is pretty trivial since it's a small text file, all we're missing is a registry for them. If we had that, we could easily implement something like:

docker-compose up --url http://compose-registry.docker.com/username/myapp.yml

which would GET the myapp.yml, and proceed with a normal up.

Publishing the app could be something like docker-compose publish, which runs docker push for all the images, and a POST for the docker-compose.yml to the url.

[thaJeztah]

A good start could be to support GitHub, similar to what docker supports with docker build; https://github.com/docker/docker/blob/master/docs/reference/commandline/build.md

E.g. docker-compose up (https://)github.com/me/myproject would clone the project in the current directory, and build/start the project

[dnephin]

I might be good to break down this problem into a few components:

• publish - how files are sent to a registry
• registry - how the files are stored/served
• consume - how to use a published file to start an app

I think a solution to each of these is should be pretty straightforward:

• publish - docker-compose publish <url> a fairly simple http POST
• registry - either docker hub, or some really simple CRUD app to receive and serve files
• consume - docker-compose up --url <url> (or something similar) http GET and proceed as usual

I think the undefined piece is "what gets published?". It's unlikely the docker-compose.yml being used in dev will be appropriate for prod. I guess #1784 covers those concerns.

[dnephin]

There are also interesting use cases regarding deployment and rollback - if you could serialise/deserialise between the state of an app and an immutable artifact,

Isn't this immutable artifact just a docker image? Roll-forward and roll-back would just be changing the image tag for each service (which ties back into the "what to publish" question) and running up.

The only time that doesn't really work is persisted state (volumes), but trying to snapshot those is probably not realistic, or at least outside the scope of what we need to do here.

Are there cases this approach doesn't cover?

[aanand]

Isn't this immutable artifact just a docker image?

The immutable artifact for the whole app would be several docker images (one per service) plus a Compose file, as I understand it. It's basically "this is what is running".

The only time that doesn't really work is persisted state (volumes), but trying to snapshot those is probably not realistic, or at least outside the scope of what we need to do here.

Yeah, I feel like it's right to leave those out.

[preflightsiren]

A use case I think hasn't been considered is how one might use compose to provide an evaluation/test environment for an application. I'm working with atlassian/stash which would be better with extra pieces (lb,storage volume, db). Currently we have to publish multiple steps for users to run the pieces they need and hope they do it correctly. something simple like docker-compose up --url http://www.atlassian.com/docker-compose/stash.yml would be perfectly useful, and would follow the pattern of docker build URL

I also think adding support for the golang project format (docker-compose up (https://)bitbucket.org/atlassian/stash-compose/stash.yml) would be great, so +1 to that

[ysokolovski]

Would like to +1 this one. This is indeed a big missing feature which would close the build-publish-deploy-run loop quite nicely. Any decent size app will be multi-container based and having the extra step of obtaining the docker-compose YAML currently complicates people's life.

[aanm]

I think it would also be useful to run also as docker-compose up foo/service-bar and also docker-compose foo/service-bar -f docker-compose.prod.yml up -d

[schmunk42]

Hey guys, you already have that...

curl https://raw.githubusercontent.com/phundament/playground/ba39a695a89206f8ecb7f13b8867a88836381db5/stacks/app-tutum/docker-compose.yml \
    | docker-compose -f - up -d

almost 😜

[ghost]

@schmunk42 i have been used something similar using a github repository and a set of scripts and was working pretty well 😋
The github repo solves the auth&auth problem in my case, giving access to certain users to this repo or handling ssh keys from servers.

[shin-]

If anyone is still watching this issue, the effort around this has moved to https://github.com/docker/app - which leverages the Compose file format.

Please have a look and give us your feedback! On the flipside, I'm going to close this as it has become out of scope for Compose itself.