add support for build secrets

[ndeloof]

What I did
add support for build-time secrets.

also let compose-go (re)build volume string to be used by engine bind API when we can't use mount

Related issue
closes #6358
closes #9380

(not mandatory) A picture of a cute animal, if possible in relation with what you did

[glours]

LGTM!
Thanks @ndeloof to add this long-awaited feature

[Wirone]

Thank you @ndeloof 🍻 When will it be available? In v2.5?

[langri-sha]

Cheers 🍻!!!

[josephlim75]

Any timeline this will be released ? Does this changes provide the ability to pass secret via docker-compose command line as well ? Example

docker compose build --secret id=xx,src=/path/secret

I am using latest version 2.4.1 docker compose, it doesn't support CLI secret argument

Build or rebuild services

Options:
      --build-arg stringArray   Set build-time variables for services.
      --no-cache                Do not use cache when building the image
      --progress string         Set type of progress output (auto, tty, plain, quiet) (default "auto")
      --pull                    Always attempt to pull a newer version of the image.
  -q, --quiet                   Don't print anything to STDOUT
      --ssh string              Set SSH authentications used when building service images. (use
                                'default' for using your default SSH Agent)

[jenciso]

Good to know. Please let me know when this feature will be released.

[pdfrod]

Apparently this has just been released in v2.5.0 🎉

Looking at the pull request it seems it only implements the ability to add secrets inside the build section of a docker-compose.yaml file (usage example here), so it doesn't look like your CLI use case is supported @josephlim75.

[bluebrown]

I think there are a few points missing:

• cli build command is missing secret option
• yaml spec is missing ssh option
• it is not possible to use a secret from an env variable
• cli up command has a build option but not way to provide ssh or secret

[ndeloof]

cli build command is missing secret option

secrets have to be defined by the secrets section, just like runtime secrets, there's no plan to offer command line flags for this purpose.

yaml spec is missing ssh option

Not sure what you mean
https://github.com/compose-spec/compose-spec/blob/master/build.md#ssh

it is not possible to use a secret from an env variable

This is tracked by compose-spec/compose-spec#251
While considered a bad practice, this is a common pattern we should address.

cli up command has a build option but not way to provide ssh or secret

both ssh and secret should be declared in compose.yaml file, just like other build attributes

[bluebrown]

If you have an ssh option on the build command, then it should have a secret option too, as is the case with docker build. Otherwise, it's inconsistent, IMO.

This flag exists for compose build:

  --ssh string              Set SSH authentications used when building service images. (use 'default' for using your default SSH Agent)

[ndeloof]

This isn't strictly consistent indeed. docker build obviously only has flags as there's no such thing like a yaml file to define parameters. I'm not against a new build --secret flag, just wonder about the use cases one could not declare an explicit entry in secrets section.

[bluebrown]

What is the usecase of all the other build options? Pretty much all of them have a respective setting in the yaml spec.

[ndeloof]

I don't understand what you mean by "have a respective setting in the yaml spec".
You can set build secrets and ssh key using your compose.yaml file, I just wonder the use-cases that require to pass those on command line. --ssh indeed has been introduced, while I was not convinced for the need, but ¯\(ツ)/¯

[Wirone]

If you have an ssh option on the build command, then it should have a secret option too, as is the case with docker build. Otherwise, it's inconsistent, IMO.

IMHO the whole idea of Compose is to have manifest file with all required stuff, which can be built and executed with bare minimum CLI commands. I don't see any sense of having docker-compose.yml and then calling docker compose --secret --volume --network etc., like in docker build. Whole infra should be defined in compose file, so everyone can just call docker compose up -d and have the exact same environment.

So I agree with @ndeloof here 👍

[ndeloof]

that's indeed the root of the compose approach, but for usability some options still are exposed to the command line, so this is all about accepting some compromises and avoid getting all options exposed as flags "just because it's possible"

[bluebrown]

I guess you are right. My points have been addressed and don't hold up. Back away slowly

[smac89]

Is it possible to get this feature into v1?

[smac89]

I have tested it and there are two issues I've noticed:

1. It is not supported for compose v1
2. It does not support the long syntax for specifying secrets. i.e. it ignores the target field, which is what determines the name of the secret when the long syntax is used

It also seems not to support loading secrets from an external source, but I guess this makes sense since external secrets may not be useful at build time...

[glours]

@smac89 Compose v1 is reaching end-of-life soon, we won't backport anything to this version and will only apply security fixes

[smac89]

@glours I kinda figured that's the case, so I've gone ahead and made the switch to the compose plugin

[tjad]

@ndeloof How does this feature work with intermediate (or multi-stage) builds ?
I have an intermediate image tag which uses the secret, however the secret is not available when I do the docker-compose configuration for the final build tag. Was this considered ? - is there another version that supports this ?

I am using docker-compose v2.5.0

Edit : This does actually work. I configured incorrectly (thought file name would be same, however the file name is the secret's name in /run/secrets

Thanks a lot! @ndeloof

[dotarj]

I tried using Docker Compose build secrets using the samples provided in the pull request but the secrets are not injected in the Dockerfile when I use docker compose build. I'm using Docker Desktop on Windows 10 (I just did a fresh install). When trying the same command using the same files in an Ubuntu VM, the secrets are injected in the Dockerfile.

What could be wrong?

docker-compose.yml

services:
  ssh:
    image: build-test-secret
    build:
      context: .
      secrets:
        - mysecret

secrets:
  mysecret:
    file: ./secret.txt

Dockerfile

FROM alpine

RUN echo "foo" > /tmp/expected
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret > /tmp/actual
RUN diff /tmp/expected /tmp/actual

secret.txt

foo

The output of docker version on the Windows 10 machine is:

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

And the output of docker compose version on the Windows 10 machine is:

Docker Compose version v2.23.0-desktop.1

[Wirone]

@dotarj I believe you should just create new issue, instead of commenting PR merged more than a year ago 😉.

[dotarj]

@Wirone Yes, I fully agree. I'll create an issue for this. Thanks for the hint. 😉

[jymchng]

@dotarj Did you manage to solve your issue and what is the issue that you created? I am having problems with trying to get my Dockerfile to 'see' the secrets defined in docker-compose.yaml.