Support for BuildKit extensions (secrets)

[tommie]

What is the problem you're trying to solve

With multi-stage builds, it's useful to be able to e.g. access an SSH server only from within the build image. Docker now supports that with BuildKit. It adds new command line flags and Dockerfile syntax. Compose should allow defining build secrets, just like it supports runtime secrets.

Building using BuildKit is enabled through an environment variable (DOCKER_BUILDKIT=1) or a Docker configuration file setting.

Compose already supports runtime secrets, which are files "mounted" inside a container. BuildKit seems very similar, but with some differences:

1. --ssh, which simply mounts a Unix socket and sets the SSH_AUTH_SOCK environment variable for RUN commands that request it.
2. The dst flag, which specifies an alternative directory (default is still /run/secrets/).
3. Though it's undocumented, it seems setting owner and mode is supported.

Describe the solution you'd like

This is a high-level discussion around the requirements of a solution. A more concrete suggestion would probably be more suited for a PR.

Tie in to the existing secrets. A file secret is defined by

• Type, but only file is currently supported
• ID (used to reference the secret in a RUN
• Source (the file path)

An SSH secret is defined by

• ID
• Agent socket paths

Both of them are referenced/dispatched by specifying

• Source (only for file secrets; implied for SSH)
• Target
• Required (boolean)
• UID
• GID
• Mode

This seems similar enough that it should be possible to extend the secrets section, and allow referencing it in an image's build section. The SSH type is specific to BuildKit, so parsing code would have to know which type is allowed in build/runtime. Using a separate section for build secrets is a possibility.

Lastly, adding a "builder" selector that allows choosing "buildkit" as the builder could set the right environment variable to allow parsing these flags. (The Dockerfile still needs to have the syntax line to support the RUN --mount variant.)

Additional context

A spec PR was suggested as the first step, in docker/compose#7296 (comment).

The --secret command line option is parsed into a github.com/moby/buildkit/session/secrets/secretsprovider.FileSource. The --ssh option is parsed into a github.com/moby/buildkit/session/sshforward/sshprovider.AgentConfig.

The secret mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_secrets.go, and the ssh mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_ssh.go.

[AkihiroSuda]

cc @tonistiigi @tiborvass

[pikeas]

This would be great, is there a timeline for when the proposal will be reviewed?

[EricHripko]

Hi @pikeas, thanks for chasing this up 👍 It's holiday season now, so a lot of people are unavailable, but I added this proposal to be discussed in the next community meeting.

[EricHripko]

To better understand the demand for each proposed function, I'd love for subscribers on this issue to vote. Could you please do the following?

• Add 🎉 to this comment if you're interested in file secrets
• Add 🚀 to this comment if you're interested in SSH secrets

Feel to add both if you're interested in both types of secrets; and thank you for your engagement 🙂

[srittau]

Docker 20.10 also supports env secrets. We use those to pass our Font Awesome auth token to docker:

DOCKER_BUILDKIT=1 \
  docker build \
  --secret id=fontawesome,env=FONTAWESOME_NPM_AUTH_TOKEN \
  .

Dockerfile:

...
RUN --mount=type=secret,id=fontawesome,required \
  export FONTAWESOME_NPM_AUTH_TOKEN=$(cat /run/secrets/fontawesome) && \
  yarn install --frozen-lockfile
...

Support for those would be useful as well.

[EricHripko]

Had a discussion about this in the Compose Spec meeting today, my notes are below:

• There was general agreement that this is one of the most requested features we have today (shout-out to the community 😉 )
• There was also a point raised that build-time secrets aren't uncommon (e.g., internal package registries that require auth)
• Primary concern was that we shouldn't make the spec too Docker-specific if possible (to enable other implementations to thrive)
• A complicating factor was that alternative Compose implementations of today typically focus on runtime aspect of workflow (as opposed to build)
• A good rule of thumb that was shared - demonstrate at least one use case in addition to local one

I've taken the action point of writing down some use cases to verify that this wouldn't be too Docker-specific. Equipped with this information, we'll be able to regroup and hopefully make the call on this proposal. Thus, if anyone wants to share various tools used to build environments (not only containers, but also things like VMs, change-roots...) please feel free to drop them in comments and I'll add it to my research materials 🙂 Thanks everyone again for their patience and positive engagement!

[tommie]

Thanks, Erik.

A complicating factor was that alternative Compose implementations of today typically focus on runtime aspect of workflow (as opposed to build)

What do you mean by "alternative implementations" here?

Given that BuildKit is newish to Docker, it seems like caring about build once runtime is working is a natural progression.

[Niek]

There's a 1,5-years old PR that solves this is a less elegant way: docker/compose#7046

IMHO it would make most sense implement build secrets the same as runtime secrets:

services:
  my-application:
    build:
      secrets:
        - my-secret

secrets:
  my-secret:
    file: /path/to/local/file

[tommie]

@Niek Agreed. I thought I made a PR for this spec change proposal, but apparently I never committed it. It's identical to yours, except, as with runtime secrets, there's a long and short form.

I saw/see no reason why this couldn't use the normal secrets mechanism, other than--possibly--secrets being loaded eagerly from secrets rather than as-needed from build/secrets. I don't remember if that's the case or not. Someone else will probably know by heart.

However, we also need the switch to enable BuildKit so the RUN --mount option is parsed in Dockerfile. Perhaps that should be controlled by requirements/capabilities, rather than "is-buildkit". Something like

services:
  my-application:
    build:
      requires:
        - run-mount

as opposed to

services:
  my-application:
    build:
      builder: buildkit

[EricHripko]

As per comment above, took a look at other tools for building environments:

• Containers: as expected, these have a number of ways to mount the secrets from the host.
  • buildkit - supports file secrets via RUN --mount=type=secret,id=mysecret ... and --secret id=mysecret,src=mysecret.txt
  • buildpacks - generically supports mounts via --volume <host path>:<target path>[:<mode>]
  • s2i (source-to-image) - generically supports mounts via --volume <host path>:<target path>[:<mode>]
  • buildah - supports explicit mounts (via --mount=type=TYPE,TYPE-SPECIFIC-OPTION[,...]) and implicit global mounts (configured via mounts.conf)
  • kaniko can only have global secrets, defined in K8S manifest
• Virtual Machines: these typically use variables (sometimes marked as sensitive) that are referenced in resource definitions. The examples are rarely file-based and are either sourced from environment variables or remote storage.
  • packer - secrets are specified as environment variables and referred by the templates via var.
  • ansible - secrets typically are specified as variables in playbooks. Variables can hold lists and maps (Proposal: hierarchical secrets #93 might be of relevance here) and can be set in variety of ways.
  • terraform - secrets are again typically specified as variables that contain strings. These can be either supplied as environment variables or encrypted in the resource definition with a dedicated KMS.

There were a few other cases (like NixOS Containers), which didn't seem to be too different from the above - so they aren't mentioned here.

Planning to bring this up at the next meeting to discuss and see if we can make the call on the proposal.

[EricHripko]

@Niek @tommie thank you for this context, this'll be really useful to help shape the schema if this proposal gets the greenlit 👍

[chris-crone]

I think that secrets (file and ssh) for build would be a useful addition.

@tommie

builder: buildkit

I don't think that we need to add a builder field to the specification. I think that most users will use the same builder for all services and will want to specify it at the tool level if anywhere.

On where to put the build secrets, I raised last night in the meeting that I like the simplicity of adding them to the existing secrets section but I worry this might confuse users. Build and deployment might take place in separate environments such that deployment and build secrets live in different places. e.g.: You might build your images on your local Docker Engine, push the images, and then deploy the workload to Kubernetes. In this case, the build secret might be a file on your disk but the deployment secrets would be a Kubernetes secret.

Thinking about this a bit more, tooling should be able to make this clear to users. e.g.:

services:
    one:
        build: .
            secrets:
                - mysecret
    two:
        image: myimg
        secrets:
            - mysecret

secrets:
    mysecret:
        file: ./secret.txt

When the user does a build, the tooling would mount the secret as expected. When the user does a deploy to Kubernetes, it would create a Kubernetes secret from the file and then mount it into service two.

[tommie]

I think that most users will use the same builder for all services and will want to specify it at the tool level if anywhere.

That's probably true, and AFAIK, BuildKit is fully backwards compatible. Whether that'll be true for new builders in the future is hard to say.

However, I think the main question here is how self-contained the docker-compose file should aim to be. Being able to say that a service (or all services in the file) require BuildKit sounds safer than having to specify that bit elsewhere. Without it, the secrets are mostly (fully?) unusable because you can't mount them (and the RUN --mount would even fail parsing). IMHO, it would be best if the Dockerfile could say that it needs BuildKit to be parsed. Second best would be docker-compose.

When the user does a build, the tooling would mount the secret as expected

So long as prod only uploads secrets that have been referenced by non-builds, I agree. (I don't know how Compose/Swarm works in that regard.) Uploading build secrets to a cluster that doesn't need them would be needlessly increasing the attack surface.

[EricHripko]

@chris-crone, do you think we'd able to simplify this proposal by treating build-time and run-time secrets as separate concepts? (I'm naively assuming that reusing build secrets during runtime isn't that common).

[chris-crone]

@chris-crone, do you think we'd able to simplify this proposal by treating build-time and run-time secrets as separate concepts? (I'm naively assuming that reusing build secrets during runtime isn't that common).

Not sure I understand. Do you mean having different sections for run and build secrets in Compose?

[EricHripko]

Basically - yes. If we're worried that mixing build/runtime secrets is confusing (per #81 (comment)), then maybe we can avoid secrets top-level attribute in favour of defining something inline under build. We could even explore going in the direction of something like bindings (although this might be too k8s-specific).

[kshcherban]

Struggling without ssh build feature in docker-compose.

[madhukar93]

Struggling without ssh build feature in docker-compose.

Yeah, we're avoiding the use of docker-compose build because of this. Any workarounds?

[tommie]

Yeah, we're avoiding the use of docker-compose build because of this. Any workarounds?

My hack was to run an ssh-agent in a separate container and use a UDS->TCP->UDS bridge using socat. Another TCP socket publishes the public key so the known-hosts works. It feels better than mounting the private key file, but the open TCP socket (on a dedicated Docker network used for builds) is obviously ugly. At least it's difficult to steal the private key from the build container. And I'm not entirely sure the TCP socket makes a difference: if I only had a UDS, it would still be open to the non-root user running the build, so it's practically the same attack surface.

It's an ugly setup, though. Security is easier the simpler the setup is. :)

[kshcherban]

Struggling without ssh build feature in docker-compose.

Yeah, we're avoiding the use of docker-compose build because of this. Any workarounds?

We had to write a script wrapper around docker-compose that parses yaml file, finds our custom properties and then builds needed images with CLI. That sucks but it's simpler than supporting docker-compose fork.

[briceburg]

I was referred here from the compose-cli / v2 project. IMO it would be most intuitive to support the --ssh build argument on parity with the docker build client and not to combine it with other secrets handling. this goes along w/ the principal of least surprise. e.g.

services:
  foo:
    build:
      context: ./src
      ssh: default

reads as a user would expect after consuming docker build documentation. supporting docker compose build --ssh default also would make sense.

this is keeping us from using docker compose to build services that rely on the ssh-agent passthrough for cloning repositories. I've detailed our ugly workaround in: docker/compose-cli#2062 (comment)

[akindo]

👍 for this feature. 🔐

[alechirsch]

We also had to write a separate shell script to use docker build directly so we can make use of the ssh injection for installing private repositories. Given how much demand for this feature is, I am surprised there is no solution here after almost two years.

[maciejjaskowski]

@kshcherban @alechirsch Would you mind sharing your scripts ?

[kshcherban]

@maciejjaskowski it's something like below, if x-build-with-docker-build is set for service in docker-compose.yml then script will process it.

yq="python3 -m yq"
BUILD_COMPONENTS=($(yq -r -e '.services | keys[]' < docker-compose.yml))

for component in "${BUILD_COMPONENTS[@]}"; do
    # shellcheck disable=SC2016
    if yq -e '.services[$ARGS.positional[0]]["x-build-with-docker-build"]' \
        --args "$component" < docker-compose.yml > /dev/null
    then
        REL_CONTEXT=$(yq -er \
            '.services[$ARGS.positional[0]].build.context' \
            --args "$component" < docker-compose.yml)
        REL_DOCKERFILE=$(yq -er \
            '.services[$ARGS.positional[0]].build.dockerfile' \
            --args "$component" < docker-compose.yml)
        TARGET=($(yq -er \
            '.services[$ARGS.positional[0]].build.target | if . then "--target", . else "" end'\
            --args "$component" < docker-compose.yml))
        TAG_WITH_VARIABLES=$(yq -er \
            '.services[$ARGS.positional[0]].image' \
            --args "$component" < docker-compose.yml)
        TAG_EXPANDED=$(bash -c "echo \"$TAG_WITH_VARIABLES\"")
        IFS=$'\n'
        DOCKERFILE_BUILD_ARGS=($(yq -r \
            '.services[$ARGS.positional[0]].build.args // []
               | to_entries[]
               | ["--build-arg", "\(.key)=\(.value)"] | .[]' \
            --args "$component") < docker-compose.yml)
        unset IFS
        BUILD_CMD=(docker build --pull --network host --ssh default \
            -f "$REL_CONTEXT/$REL_DOCKERFILE" \
            -t "$TAG_EXPANDED" \
            "${TARGET[@]}" \
            "${DOCKERFILE_BUILD_ARGS[@]}" "${BUILD_ARGS[@]}" "${REL_CONTEXT}")
        echo "* Running: ${BUILD_CMD[@]}"
        "${BUILD_CMD[@]}"
    fi
done

[maciejjaskowski]

@kshcherban oh wow, thank you! One noob question: how do you pass the --secrets flag ? Or is it something that you left out as an easy exercise ?

[kshcherban]

@kshcherban oh wow, thank you! One noob question: how do you pass the --secrets flag ? Or is it something that you left out as an easy exercise ?

@maciejjaskowski we don't use secrets, but you can include it in script using the same approach

[basicdays]

Just want to point out the use case that I'm most commonly in. For things like getting access to things like Github Packages npm registry, I'd need a token to pull packages from there. I have a token value defined in the environment on CI during the docker build step.

Lets say I have a env var SECRET_TOKEN I'd want to use to auth to a registry. Currently with docker buildx bake, I just have something like the following to pull the secret into the build:

services:
  app:
    image: the-image:build-id
    build:
      context: ./
      x-bake:
        secret:
          - id=SECRET_TOKEN

I kind of see them more analogous to how bind mounts are defined in docker-compose. that would at least allow them to be defined in-line instead of in the top-level secrets section used for containers.

[jheld]

What are the blockers here to implementing?

[glours]

@jheld feel free to provide feedback on the #238 PR and compose-spec/compose-go#236

[ndeloof]

Closing as implemented

[Niek]

The relevant docs link, in case anyone is looking for it:

https://docs.docker.com/compose/compose-file/build/#secrets