ARG before FROM in Dockerfile doesn't behave as expected

[Benjamin-Dobell]

Description

It's documented that ARG can appear before FROM, so that arguments may be substituted into image names etc.

Rather than having some ARG before and some ARG after FROM, for consistency I attempted to place all my ARG before FROM. However, to my surprise (after a lot of debugging) I determined that my arguments are always blank after FROM.

I believe the meta-arg functionality/refactoring may somehow be responsible:

239c53b

Steps to reproduce the issue:

1. Produce a Dockerfile such as:

ARG environment
FROM alpine:3.5
ENV ENVIRONMENT=${environment:-development}
RUN echo "$ENVIRONMENT" > /value_of_environment

2. Build the image and run the image, printing the value of environment ARG (stored in /value_of_environment):

docker run $(docker build -q --build-arg environment=production .) cat /value_of_environment

Describe the results you received:

development

Describe the results you expected:

production

Additional information you deem important (e.g. issue happens only occasionally):

Altering the Dockerfile such that ARG comes after FROM i.e.

FROM alpine:3.5
ARG environment
ENV ENVIRONMENT=${environment:-development}
RUN echo "$ENVIRONMENT" > /value_of_environment

then running again:

docker run $(docker build -q --build-arg environment=production .) cat /value_of_environment

gives the expected output of production.

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:31:53 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 59
 Running: 0
 Paused: 0
 Stopped: 59
Images: 370
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 457
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.31-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.818GiB
Name: moby
ID: BCV5:MEMK:BYKI:I2IU:QY2V:5DRM:F2FP:JFAG:SM46:M2WJ:73YV:3KLP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 20
 Goroutines: 40
 System Time: 2017-07-16T19:58:09.054157098Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[boaz0]

@thaJeztah correct me if I'm wrong.

@Benjamin-Dobell after investigating this, 239c53b is not the origin of this behavior.

Basically, after the FROM instruction all the build arguments are reset and thus aren't available in the Dockerfile.

From what I found the purpose of ARG before FROM is to use it inside the FROM instruction #31352

[thaJeztah]

Yes, this doesn't look like a bug; see this pull request, which adds some more information docker/cli#333

[boaz0]

@thaJeztah I guess we can close this

[Benjamin-Dobell]

Irrespective of whether this was implemented this way intentionally or it's a bug; I think it's a bit of a usability nightmare.

It's not clearly documented that this is the expected behaviour, and it makes for messy Dockerfile. But more importantly, it opens a pandora's box of confusing edge-cases.

What if I intend to use an ARG in both my FROM statement and after it? Am I expected to have multiple ARG statements referring to the same build-arg?

What happens if I use default value syntax ARG argument=some_value before FROM and just ARG argument after FROM? What is the expected value of argument after FROM if no argument build-arg was passed?

[thaJeztah]

What is the expected value of argument after FROM if no argument build-arg was passed?

The same as it would be if you're not using multi-stage build; empty / no value set

[Benjamin-Dobell]

@thaJeztah I know that's true now, I've experimented with it. The issue is that it's hugely non-obvious.

If this is expected behaviour and no-one is willing to change it. Then at the very least ARG ought to be deprecated (before FROM) and instead when used prior to FROM the syntax should be FROMARG (which must come before FROM).

[thaJeztah]

ARG is reset after each FROM. If this is documented; why would ARG before FROM have to be deprecated?

/cc @tonistiigi @dnephin

[Benjamin-Dobell]

Improved documentation is always appreciated, and would have saved me some time. However, just because behaviour is documented doesn't preclude the behaviour itself from scrutiny.

ARG has too much complexity to it. I'd argue this functionality shouldn't have been added to the ARG keyword in the first place, it's effectively been repurposed and its behaviour is now far to nuanced. A new keyword FROMARG from the on-set would have made a lot more sense.

[Benjamin-Dobell]

I should note, that I'm not actually an advocate of expanding the grammar when the usage of the existing grammar can be expanded.

However, in this particular instance ARG has had its existing semantics altered; the behaviour is not additive. Previously whenever you referenced an ARG defined argument you'd have access to the value as expected. Now argument interpolation is much more context aware.

It's extremely confusing in single stage builds, and perhaps more-so in multi-stage ones. If arguments really are tied to build stages (although I must confess I'm not sure why this is desirable), then you've suddenly a need to look at the previous "stage", beyond the FROM verb.

Realistically, you can't pass different arguments to different build stages (they're typically provided as CLI arguments). So there's no legitimate reason to scope arguments to build stages. Additionally:

a “cache miss” occurs upon its first usage, not its definition

So there is zero incentive to intersperse ARG definitions through-out a file. Therefore, the most logical behaviour would be to encourage all ARG definitions to be placed at the top of a file (where they can clearly be seen) and then update the behaviour to ensure there's no funny business with build stages.