builder: Environment variables in CMD are not used

[jtagcat]

Steps to reproduce the issue:
In Dockerfile:

ENV FOOBAR=fish
CMD ./$FOOBAR

and

ARG FOOBAR
CMD ./${FOOBAR}

and

ENV FOOBAR=fish
CMD [ "./${FOOBAR}" ]

None of these work.

Describe the results you received:
Variable is empty (CMD ./).

Describe the results you expected:
Variable is used (CMD ./fish).

[thaJeztah]

This is (currently) the expected behavior, but documentation can definitely be improved, as this is somewhat "complicated", and can be confusing (also see docker/cli#3323).

Docker itself (docker build) does not perform environment variable substitution in CMD, ENTRYPOINT and RUN commands (see https://docs.docker.com/engine/reference/builder/#environment-replacement). Environment variables in those commands are handled by the shell (unless the JSON / "exec form" syntax is used), which means that those variables are evaluated the moment the shell is executed.

• For RUN, this means: the moment when the RUN command is executed as part of the build
• For CMD and ENTRYPOINT this means: after the image has been built, and when the container is started

First example

Based on your first example:

docker build -t repro-42937 -<<'EOF'
FROM alpine
ENV FOOBAR=fish
CMD echo $FOOBAR
EOF

The ENV persists the value of the environment variable in the image

docker image inspect --format='{{json .Config.Env}}' repro-42937
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","FOOBAR=fish"]

But the $FOOBAR variable in CMD has not (yet) been evaluated, so the CMD of the image is:

docker image inspect --format='{{json .Config.Cmd}}' repro-42937
["/bin/sh","-c","echo $FOOBAR"]

The variable will be evaluated when the command is executed, which also allows you to override the env-variable at runtime:

docker run --rm repro-42937
fish

docker run --rm --env FOOBAR=turtles repro-42937
turtles

Second example

In your second example, FOOBAR is set as a build-arg:

docker build -t repro-42937-2 -<<'EOF'
FROM alpine
ARG FOOBAR=fish
CMD echo $FOOBAR
EOF

Unlike ENV, build-args are not persisted in the image, and only available during docker build. This means the image does not have a FOOBAR environment variable set by default:

docker image inspect --format='{{json .Config.Env}}' repro-42937-2
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]

But (like the first example), the $FOOBAR variable in CMD has not (yet) been evaluated, so the CMD of the image is:

docker image inspect --format='{{json .Config.Cmd}}' repro-42937-2
["/bin/sh","-c","echo $FOOBAR"]

The variable will be evaluated when the command is executed, which also allows you to override the env-variable at runtime. As the image does not have a FOOBAR environment variable set by default, the default command will print an empty string:

docker run --rm repro-42937-2

But if you specify the environment variable at runtime, it will be printed:

docker run --rm --env FOOBAR=turtles repro-42937-2
turtles

Third example

Your third example is simlar to the first one, however, this time you're using the "exec" form (JSON). The "exec" form is used to execute a command without using a shell (/bin/sh by default). This is useful for situations where you either do not want the command to be run as a sub-process of a shell, or if (for example in a FROM scratch image), no shell is available in the image.

As explained above, environment substitution in CMD, RUN and ENTRYPOINT is performed by the shell. In the "exec" form, no shell is present, so no environment variable substitution will be performed, unless you explicitly set a shell to execute the command (I'll put an example further below);

docker build -t repro-42937-3 -<<'EOF'
FROM alpine
ENV FOOBAR=fish
CMD ["./${FOOBAR}"]
EOF

Inspecting the image shows;

docker image inspect --format='{{json .Config.Cmd}}' repro-42937-3
["./${FOOBAR}"]

Running the image will produce an error, because (without shell), the env-var is not substituted, and ./${FOOBAR} is not a valid command;

docker run --rm repro-42937-3
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380:
starting container process caused:
exec: "./${FOOBAR}": stat ./${FOOBAR}: no such file or directory: unknown.

Changing the Dockerfile to explicitly set a shell to execute the command (I'm changing it to an echo to illustrate) will make the command work;

docker build -t repro-42937-4 -<<'EOF'
FROM alpine
ENV FOOBAR=fish
CMD ["/bin/sh", "-c", "echo $FOOBAR"]
EOF

This makes the image the same as the first example:

docker image inspect --format='{{json .Config.Cmd}}' repro-42937-4
["/bin/sh","-c","echo $FOOBAR"]

By default it will use the FOOBAR env-var as defined in the image, which, again, can be overridden at runtime;

docker run --rm repro-42937-4
fish

docker run --rm --env FOOBAR=turtles repro-42937-4
turtles

But what about RUN ?

Things can be confusing when using environment variables in RUN, because environment variables seemingly are substituted. For example, take the following Dockerfile:

FROM alpine
ENV FOO=hello
ARG BAR=world
RUN echo FOO is $FOO and BAR is $BAR

Building the above Dockerfile

docker build --no-cache --progress=plain .
...
#4 0.212 FOO is hello and BAR is world
...

The reason this works is because (as outlined above), variable substitution is handled by the shell that's executing the RUN. docker build will set all environment variables and build-arguments that are set in the Dockerfile when executing the RUN command, after which the shell handles those. (note there's a bug / issue in the BuildKit output, which wrongfully does show the commands with env-vars substituted; opened a ticket for that: moby/buildkit#2415)

Basically, the above RUN instruction is executed as below:

FOO=hello BAR=world /bin/sh -c 'echo FOO is $FOO and BAR is $BAR'

Shell commands running in a RUN can therefore also access environment variables that are defined in the shell, but unknown to Docker / docker build, such as $HOME;

FROM alpine
ENV FOO=hello
ARG BAR=world
LABEL somelabel="FOO is $FOO and BAR is $BAR, but HOME is $HOME"
RUN echo FOO is $FOO and BAR is $BAR, but HOME is $HOME

Building the above, you can see that, while $HOME is unknown to docker build itself, the shell knows about this variable, and can use it;

docker build -t foo --no-cache --progress=plain .
...
#5 0.208 FOO is hello and BAR is world, but HOME is /root
...

But inspecting the image shows that $HOME is not set in the label, and was substituted by an empty string;

docker image inspect --format '{{json .Config.Labels}}' foo
{"somelabel":"FOO is hello and BAR is world, but HOME is "}

[jtagcat]

Thanks for your (much) detailed response.

@thaJeztah then, this issue is dupe of docker/cli#3323?

I went for the ["/bin/sh","-c","./$FOOBAR"], but I'd still prefer an explicit buildtime-only solution. Though, besides documentation, would a breaking change be too much of an unneeded effort?

[jtagcat]

Personally, it'd make sense to have the following vars:

• evaluated at (only) build time
• eval at both build time and runtime; latter overwriting the former, if set
• not evaluated by docker at all, evaluated by shell (current)

[thaJeztah]

Thanks for your (much) detailed response.

Thank you! This topic really needs better documentation / examples; some time should be spent on that (and we'd need a good place for it in the documentation - perhaps separate from the general Dockerfile syntax page - to allow for it to be described in-depth).

Personally, it'd make sense to have the following vars:

Yes, in hindsight (my personal 0.02c - I had some discussion on this with others, and there was no real consensus 😅), perhaps there should've been a separate syntax for variables "known by the builder". Something like;

FROM busybox
ARG SOME_ARG=hello
ENV SOME_ENV=hello
RUN echo %{SOME_ARG} $SOME_ENV 
LABEL my_label=%{SOME_ARG}

Or even:

FROM busybox
ARG SOME_ARG=hello
ENV SOME_ENV=hello
RUN echo %{arg:SOME_ARG} %{env:SOME_ENV}
LABEL my_label=%{arg:SOME_ARG}

This would allow distinguishing them better, and clearer what substitutions are "templating" / performed by the Dockerfile parsing, and which ones are evaluated by the shell, although it would have to be looked at if that could cause additional confusion, e.g. both variants below would result in the same, but for different reasons:

RUN echo %{arg:SOME_ARG} %{env:SOME_ENV}
RUN echo $SOME_ARG $SOME_ENV

It would potentially avoid ambiguity in some cases though, e.g.:

RUN export SOME_ARG=hello echo %{arg:SOME_ARG}

At the time this was all implemented, the intent was to make these act more "natural"; env-var substitution is a concept known by most Linux users, and has a well-defined syntax; using this syntax made it organically fit into the RUN instructions without having to learn a new, dockerfile-specific syntax. I think that part is still a "pro" of the current approach, but comes with the downsides discussed here. At the time, Windows containers were not in sight yet as well, so Windows-specific syntax inside RUN commands (RUN echo %SOME_ENV%) wasn't a problem (I know this can be cause of confusion for some as well).

[jtagcat]

(and we'd need a good place for it in the documentation - perhaps separate from the general Dockerfile syntax page - to allow for it to be described in-depth).

https://docs.docker.com/engine/reference/builder/#environment-replacement ?

perhaps there should've been a separate syntax for variables "known by the builder".

I agree and disagree:

• it might make it more readable for people who know
• it might force people who don't know as well to hold the docs open while reading

Instead of % as a var prefix, I'd propose $$ for builder-specific vars. My personal favorite character not used anywhere, yet available everywhere is ¤ (perfect for quick and dirty csv!)

$ may then be used for builder-runtime, and \$ (escaped) for shell eval.

It'd probably be well readable and understandable for the mentioned 'Linux' audience.

The con here is the major breaking change. Dockerfiles don't (they probably should, like docker-compose) have a VERSION 1.2-VERSION 1 definition.

[thaJeztah]

https://docs.docker.com/engine/reference/builder/#environment-replacement ?

This topic may be quite long, so a bit in doubt if it should be put on that page (which is already lengthy), or instead a separate page (which could be linked from there)

I'd propose $$ for builder-specific vars.

I'm not tied to the syntax (so it's just a rough idea); I did try to avoid using $ as part of it, as processing may get more complicated (e.g. code performing substitution may replace $$VAR with $<value of var>).

The con here is the major breaking change.

Yes, we can't make breaking changes, so if this is to be considered, it should be backward compatible (existing ways of handling substitution should continue to work), hence a different syntax for substitutions that are handled when parsing the Dockerfile.

Dockerfiles don't (they probably should, like docker-compose) have a VERSION 1.2-VERSION 1 definition.

Note that (some years after my initial proposal #4907) Dockerfiles, when using BuildKit, now do support versioning through the # syntax directive. For example,

# syntax=docker/dockerfile:1.3

When using the syntax directive, the specified Dockerfile syntax is pulled as an image from Docker Hub (which allows any version of Docker with BuildKit enabled to use the syntax). The syntax directive is actually more flexible than just a version, as it allows for alternative front-ends, which could have a fully different syntax (e.g. see https://matt-rickard.com/building-a-new-dockerfile-frontend/)

they probably should, like docker-compose

Heh, except the docker-compose spec (https://github.com/compose-spec/) dropped versioning altogether (🙊), but that's another topic