ARG persists across build stages with buildkit, deviating from default docker behavior

[Frederik-Baetens]

I seem to have found differing behavior between buildkit & docker when building:

FROM alpine AS base
ARG HI="hi"
RUN >&2 echo $HI
from base as second
RUN >&2 echo $HI

I get differing behavior between default docker & buildkit. Default seems to be in line with the documentation, while buildkit seems to deviate.

~/Programmeren/build_test : DOCKER_BUILDKIT=1 docker build --no-cache --progress plain .
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.4s#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 96B done
#1 DONE 0.3s#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.0s#4 [base 1/2] FROM docker.io/library/alpine
#4 CACHED#5 [base 2/2] RUN >&2 echo hi
#5 0.671 hi
#5 DONE 0.9s#6 [second 1/1] RUN >&2 echo hi
#6 0.788 hi
#6 DONE 1.0s#7 exporting to image
#7 exporting layers
#7 exporting layers 0.3s done
#7 writing image sha256:7df7fdfbf51284fb9a3cd2e85b81c0038e3ad2817523da8e0aeb9271dcd4cac4 done
#7 DONE 0.3s
~/Programmeren/build_test : DOCKER_BUILDKIT=0 docker build --no-cache --progress plain .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM alpine AS base
 ---> d6e46aa2470d
Step 2/5 : ARG HI="hi"
 ---> Running in 0141b2a09d17
Removing intermediate container 0141b2a09d17
 ---> 4694317b7ccd
Step 3/5 : RUN >&2 echo $HI
 ---> Running in 3fc91617013d
hi
Removing intermediate container 3fc91617013d
 ---> 3db785731887
Step 4/5 : from base as second
 ---> 3db785731887
Step 5/5 : RUN >&2 echo $HI
 ---> Running in 9073cb75c114Removing intermediate container 9073cb75c114
 ---> 04955f9cbeed
Successfully built 04955f9cbeed

As you can see, with buildkit, the ARG is still set in the second stage, while with regular docker build, it is not set in the second stage (in accordance with the documentation AFAIK).

Relevant docs: https://docs.docker.com/engine/reference/builder/#scope

[qhaas]

I encountered this unexpected buildkit behavior and interpreted the documentation the same way as the OP when I attempted to build a Dockerfile from another FOSS project without buildkit. Thankfully, it failed since the command in RUN required the variable to be set, but I imagine if more developers assume buildkit's way of handling ARG scope in multi-stage builds is the 'correct' way, that can lead to unexpected behavior for those not using buildkit.

For example, consider the consequences of the difference between standard docker builds and buildkit builds with this dockerfile in Docker v19.03:

FROM alpine AS base
ARG APP_NAME=myapp
RUN echo "creating directory '/usr/local/${APP_NAME}' in base stage" && mkdir /usr/local/${APP_NAME}
from base as second
RUN echo "removing directory '/usr/local/${APP_NAME}' in base stage" && rm -rf /usr/local/${APP_NAME}

[mello7tre]

Same problem here, dunno why no one take in account this.
It create problems when migrating to new docker version that use by default buildkit.
I spent lot of time before finding out the source of the error that i was getting while building a container that was previously working.
This is really a subtle nasty bug (if is a feature it's not documented anywhere) and would be nice if someone put some effort to solve this.

Thanks.

[mello7tre]

Temporary fix i found out is to add the variable that i do not want to be propagated to the "child" stage to build-arg with and empty value.
Ex:

FROM base as builder-modules
ARG APPDIR
ARG NODE_ENV=production
WORKDIR ${APPDIR}
COPY package*.json yarn.lock ./
RUN yarn install

FROM builder-modules as builder-app
RUN yarn install --force

Using buildkit inside builder-app variable NODE_ENV have value production; if i want to have the same behaviour of "default/old" docker where NODE_ENV is not defined inside builder-app stage, i need to add do build command:
--build-arg=NODE_ENV=

But all this make no sense and is clearly a bug that should be fixed.