Proposal: make images (and "build") a first-class citizen in the compose spec

[thaJeztah]

I've been meaning to post this proposal (had some notes for this for some time), so decided to post my draft here.

The compose-spec currently allows users to define how to build an image for a service using the fields in the services.{service-name}.build struct.

While this works for most situations, it's not always ideal:

• Building an image using a compose file requires a service to be defined under which to put the build instructions. This does not allow for a compose-file to be used just for building images. While it's possible to use a docker-bakefile.hcl for this, using .hcl is a more "advanced" (and complex) approach.
• If multiple services are using the same image, the build options have to be either specified on all of those services (requiring docker compose build to be smart enough to de-duplicate the builds), or to be specified under a single (or "dummy") service, requiring the user to first build that service, then deploy/update the stack.
• Nesting the build-options under a service can be somewhat confusing. For example, both the service itself and the build can specify labels and network, which can confuse users.

My proposal is to add a top-level images section to the specification. This section is similar to (e.g.) networks and volumes; the images section defines how an image is created (equivalent of volumes define the definition of a volume), after which services can "consume" / "reference" the image to be used.

Having a separate section that describes images allows for:

• A compose file to only specify images (which can be useful to automate building a number of images). Consider this a bake-file "light" for situations where the advanced options provided by a .hcl bakefile may not be needed.
• A single image to be shared by multiple services.
• Adding more build options (fields), e.g. "secrets", without the ambiguity of having equally named options under the service.

In the example below, the compose spec defines three images (frontend_image, backend_image, debug_image), of which the backend_image is used for both the backend and api services. The debug_image (for illustration of possible enhancements) "extends" the backend_image, using the same build options, but a different target:

services:
  frontend:
    image: frontend_image
    ports:
      - "8080:80"
  backend:
    image: backend_image
    command: ["serve-backend"]
  api:
    image: backend_image
    command: ["serve-api"]
  console:
    image: debug_image

images:
  frontend_image:
    context: ./frontend/
    dockerfile: ./frontend/Dockerfile
    args:
      - "VERSION"
      - "HTTPS_PROXY"
    labels:
      - "org.opencontainers.image.version=$VERSION"
      - "org.opencontainers.image.revision=$GIT_COMMIT"
    target: dev
    tags: 
      - "myproject/frontend:latest"
      - "myproject/frontend:${VERSION}"
  backend_image:
    context: ./backend/
    dockerfile: ./backend/Dockerfile
    target: dev
    labels:
      - "org.opencontainers.image.version=$VERSION"
      - "org.opencontainers.image.revision=$GIT_COMMIT"
    tags:
      - "myproject/backend:latest"
      - "myproject/backend:${VERSION}"
  debug_image:
    extends: backend_image
    target: debug
    tags:
      - "myproject/debug:latest"
   

Note that tags could also be split into a local name for the image (when running the stack locally), and push (if image should be pushed to a registry after building). For example, the following would produce a local image projectname_my_frontend_image when running docker compose up (or docker compose build), but can be pushed to a registry as docker.io/myorg/frontend:latest and docker.io/myorg/frontend:$VERSION when running docker compose build --push:

images:
  frontend_image:
    context: ./frontend/
    dockerfile: ./frontend/Dockerfile
    args:
      - "VERSION"
      - "HTTPS_PROXY"
    labels:
      - "org.opencontainers.image.version=$VERSION"
      - "org.opencontainers.image.revision=$GIT_COMMIT"
    platforms:
      - "linux/amd64"
      - "linux/arm64"
      - "linux/ppc64le"
    target: dev
    name: "my_frontend_image"
    push:
      - "docker.io/myorg/frontend:latest"
      - "docker.io/myorg/frontend:${VERSION}"

If we consider "non-image" artifacts to be useful, we could also consider a top-level build or artifacts section. This could allow for (e.g.) binaries to be created instead of images, which would be mostly useful for "build-only" compose files.

A quick example of what this might look like:

artifacts:
  frontend_image:
    output: "image"
    context: ./frontend/
    dockerfile: ./frontend/Dockerfile
    args:
      - "VERSION"
      - "HTTPS_PROXY"
    labels:
      - "org.opencontainers.image.version=$VERSION"
      - "org.opencontainers.image.revision=$GIT_COMMIT"
    platforms:
      - "linux/amd64"
      - "linux/arm64"
      - "linux/ppc64le"
    target: dev
    tags: 
      - "myproject/frontend:latest"
      - "myproject/frontend:${VERSION}"
  binaries:
    extends: frontend_image
    output: "local"
    platforms:
      - "local"
    copy: 
      src: "/bin/mybinary"
      target: "./bin/mybinary"

[EricHripko]

I like this proposal for 2 reasons:

• It potentially unblocks progress towards build secrets
• It molds the existing practice into a portable format to define what it takes to build an image

For the latter, I am not sure if this duplicates docker-bake.hcl but I've not really seen many cases of buy-in there. Personally, I've found myself documenting build args/targets/etc in other sources (e.g., Jenkinsfile when building & publishing an image in CI). I wonder if someone who had a hand in docker-bake.hcl could voice their opinion on using Compose in a similar way 🙂