Flexible images with ONBUILD and ENV

[rijnhard]

Hi

According to the docs ENV is not supported with ONBUILD commands in dockerfiles.

The use case is to allow a child images to configure some options in the parent is so that you can reduce boilerplate and allow base images to be standalone and inheritable (as a template).

example:

ENV PROJECT_NAME
ENV PORT 80

RUN mkdir -p                 /opt/enigma/$PROJECT_NAME/server
RUN ln -s                    /opt/enigma/$PROJECT_NAME/ /$PROJECT_NAME
COPY scripts/*.sh            /opt/enigma/$PROJECT_NAME/
COPY scripts/$PROJECT_NAME-vhost.conf  /etc/httpd/conf.d/
RUN sed -i "s#\$PORT#$PORT#" /etc/httpd/conf.d/$PROJECT_NAME-vhost.conf 
RUN chmod +x                 start.sh

VOLUME /$PROJECT_NAME

EXPOSE 22 $PORT 443 9000

There are 4 options here:

1. build the base image first with the correct -e parameters and then target that build in your child dockerfile
2. copy that boilerplate and change it in each child dockerfile, also removing it from the parent, which effectively makes the parent useless on its own.
3. allow ENV substitution in ONBUILD which will allow the template to be used as is by specifying a project name, or as a parent by overriding the PROJECT_NAME env variable.
4. extract most of the boilerplate into a shell script and explicitly call it with parameters in the child dockerfile, but that still leaves EXPOSE and VOLUME that need to be called in the child dockerfile.

None of these are ideal, but do work as workarounds, the issues are:

1. 2 stage build process (error prone if you aren't careful, makes development environments annoying)
2. duplication of code
3. ideal - not possible
4. bypasses the point of the dockerfile IMHO.

Disclaimer: this is purely a usability concern, in that i haven't thought of the technical implications.

[duglin]

Try using ONBUILD env ... I think you'll find that it does work. I'm not sure about your particular use case but ENV is supported with ONBUILD.

[rijnhard]

@duglin in this case it's more "ONBUILD instructions are NOT supported for environment replacement, even the instructions above." (from the docs)

[duglin]

Can you show me an example of what you want your ONBUILD to look like? I'm not clear on what exactly you want to do and where you want the env substitution to happen.

[rijnhard]

Dockerfile 1: (as dev:dev)

FROM centos:centos6

MAINTAINER R H <blah@blah.co.za>

ONBUILD ENV PROJECT_NAME default
ONBUILD ENV PORT 80

ONBUILD RUN mkdir -p                 /opt/enigma/$PROJECT_NAME/server
ONBUILD RUN ln -s                    /opt/enigma/$PROJECT_NAME/ /$PROJECT_NAME
ONBUILD COPY scripts/*.sh            /opt/enigma/$PROJECT_NAME/
ONBUILD COPY scripts/$PROJECT_NAME-vhost.conf  /etc/httpd/conf.d/
ONBUILD RUN sed -i "s#\$PORT#$PORT#" /etc/httpd/conf.d/$PROJECT_NAME-vhost.conf 
ONBUILD RUN chmod +x                 /opt/enigma/$PROJECT_NAME/*.sh

ONBUILD VOLUME /$PROJECT_NAME

ONBUILD EXPOSE 22 $PORT 443 9000

Dockerfile 2:

FROM dev:dev

ENV PROJECT_NAME newproject

MAINTAINER R H <blah@blah.co.za>

WORKDIR  /opt/enigma/$PROJECT_NAME/
CMD ./dev.sh && ./start.sh

This is the idea. At the moment the ENV PROJECT_NAME may be correctly set in the child dockerfile, but in the parent onbuild commands it was already evaluated as default instead of as newproject

[duglin]

ok - even if we did do env var substitutions in ONBUILD cmds I still don't think you'd get what you wanted. The ONBUILD cmds are run before anything in the "Dockerfile 2" is processed. Which means your setting of PROJECT_NAME to "newproject" will never override the PROJECT_NAME used in the ONBUILD/COPY commands.

What you want, I think, is one (or combination) of: #9176 and #12749 . Then you could either set an env var from the docker build command line, or set PROJECT_NAME to "newproject" and the INCLUDE your base image build steps.

But as of now the way ONBUILD is designed I don't think it can ever give you want you want - alone.

[rijnhard]

mmm #12749 could well fullfil the same use case.
Not exactly, but close enough.

The simplified version of what I tried to demonstrate is actually best described as a template, in that there is a sequence of steps defined, and depending on the variables there are some variations in steps.

That could be a different feature request, I just thought since ONBUILD already delays the execution of the steps to a child image build, that it makes sense to just use that mechanism which already exists to implement a templating functionality of sorts.

[duglin]

@WiR3D-kNiGhT I think we need to close this for now due to: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

[packageman]

@rijnhard Maybe this approach can solve your problem.

In parent image, add instructions like below:

ONBUILD ARG REPO_PATH
ONBUILD ENV REPO $REPO_PATH
ONBUILD ADD . /app/$REPO

And build child image with build-arg option (docker build --build-arg REPO_PATH="xxx").

[JeanMertz]

Was anything ever done to change this behaviour?

I'm trying to get something like this to work without having to use command line arguments:

first (shortened) Dockerfile, shared between multiple projects:

FROM golang:1.10-alpine
ENV LIBRDKAFKA_VERSION=v0.11.3
ONBUILD RUN cd $(mktemp -d) \
 && curl -sL "https://github.com/edenhill/librdkafka/archive/$LIBRDKAFKA_VERSION.tar.gz" | \
    tar -xz --strip-components=1 -f - \
 && ./configure \
 && make -j \
 && make install

And then in a second Dockerfile:

FROM firstdockerfile
ENV LIBRDKAFKA_VERSION=4e7a46701ecce7297b2298885da980be7856e5f9
COPY . .
RUN go build -o /tmp/run -tags static_all

FROM scratch
ENTRYPOINT ["/run"]
COPY --from=0 /tmp/run /run

The problem (still) is that overwriting the ENV var in the second (project-specific) Dockerfile isn't working as expected. I've tried several different solutions, basically a matrix between (ONBUILD)[ENV|ARG], but none seems to work.

I know a solution is to simply use ARG and use the CLI arguments, but in our automated pipeline, that's (currently) not an option, so I'm trying to avoid going that route.

[thaJeztah]

@JeanMertz it's possible

The trick is to add an ARG before the first FROM, and give it a default value. An ARG before the first FROM is basically not tied to a "build-stage", which makes it a "global" option. However, a build-stage that wants to use it, still has to define the ARG.

First, an example without ONBUILD (just to illustrate the concept);

# Defined before the first FROM, and given a default value
ARG REPO_PATH=this-is-the-default-value

FROM busybox

# You're now in a new build-stage, so the `REPO_PATH` arg is ignored, unless you define one here.

# Define the `ARG`, but don't give it a default value: if no default-value is set,
# the  value is set to what's currently set for `REPO_PATH`
ARG REPO_PATH

# A command that uses $REPO_PATH
RUN echo $REPO_PATH > /foobar

# Setting the `CMD` to show the result at runtime :-)
CMD echo "the content of foobar is:"; cat /foobar

Building the above image (without --build-arg):

docker build -t example .

And running it, shows that the default value is used:

docker run --rm example

the content of foobar is:
this-is-the-default-value

However, you can override the value at build-time:

docker build -t example --build-arg REPO_PATH=this-is-passed-during-build

And running the image shows that the --build-arg value is now used:

docker run --rm example

the content of foobar is:
this-is-passed-during-build

Using the above with ONBUILD

With the information above, you can do the same for an ONBUILD image:

FROM busybox

# This will define the `ARG` inside the build-stage (it will be executed after `FROM`
# in the child image, so it's a new build-stage). Just like the example without `ONBUILD`,
# don't set a default value so that the value is set to what's currently set for `REPO_PATH`
ONBUILD ARG REPO_PATH

# Only needed if you want to persist the value that was used in the image's `env`
ONBUILD ENV REPO_PATH=$REPO_PATH

# A command that uses $REPO_PATH
ONBUILD RUN echo $REPO_PATH > /foobar

# Setting the `CMD` to show the result at runtime :-)
# Note that this could also be defined without using `ONBUILD` (because other
# images extend using `FROM`)
ONBUILD CMD echo "the content of foobar is:"; cat /foobar

Build the above image, and tag it example:onbuild:

docker build -t example:onbuild .

In the second Dockerfile (for the "child" image), add an ARG before the first FROM, and give it a default value, same as was done in the example without ONBUILD:

ARG REPO_PATH=hey-I-am-the-default
FROM example:onbuild

Build the child image (without --build-arg):

docker build -t example:child1 .

Running it, shows that the default value is used:

docker run --rm example:child1

the content of foobar is:
hey-I-am-the-default

And just like the non-obuild example, you can override the default during build:

docker build -t example:child2 --build-arg REPO_PATH=hey-I-was-passed-at-build-time .

Running that one shows that the value that was passed through --build-arg is used:

docker run --rm example:child2

the content of foobar is:
hey-I-was-passed-at-build-time

[JeanMertz]

Thank you for that detailed explanation @thaJeztah, this is definitely what I was looking for 👍

I must admit, the semantics of Dockerfile configurations is becoming more and more difficult to understand as time moves on and Docker gets new functionality, but at least I now know it's a possibility to achieve what I want, which is great.

[thaJeztah]

Yes, I think the docs can use some more in-depth examples. There are a lot of options available, but not all of them are "obvious". Here's one that shows how to do an "if/else" like construct; https://gist.github.com/thaJeztah/a4beecff53fefb824ec253a609c297e6

[JeanMertz]

@thaJeztah I've managed to get this working as expected, so thanks for that! There's one more lingering question:

You mentioned in the parent image:

# don't set a default value so that the value is set to what's currently set for `REPO_PATH`
ONBUILD ARG REPO_PATH

Is there any way to do set a default value in the parent container? Using REPO_PATH=... doesn't work, as it won't get overwritten using your technique. I guess I could do some shell-style checking to see if the variable is empty, and if so set the default value there, but maybe there's a more Docker'ish way of doing this?

Another solution would be to use your (unknown to me!) if/else conditional logic to make this work, I guess.

[thaJeztah]

I think there's some options to do that

(I'm using HEREDOC notation in the examples below, because I'm lazy and don't want to write the Dockerfiles to a file 😂)

docker build --no-cache -t example:onbuild -<<'EOF'
FROM busybox

# This defines the `ARG` inside the build-stage (it will be executed after `FROM`
# in the child image, so it's a new build-stage).Don't set a default value so that
# the value is set to what's currently set for `REPO_DEFAULT`
ONBUILD ARG REPO_DEFAULT

# If REPO_DEFAULT is set/non-empty, use it, otherwise use a default value
ONBUILD ARG REPO_PATH=${REPO_DEFAULT:-no-value-set-this-is-the-default}

# A command that uses $REPO_PATH
ONBUILD RUN echo $REPO_PATH > /foobar

# Setting the `CMD` to show the result at runtime :-)
# Note that this could also be defined without using `ONBUILD` (because other
# images extend using `FROM`)
ONBUILD CMD echo "the content of foobar is:"; cat /foobar
EOF

Not setting the default in the child image, will use the default value:

docker build --no-cache -t example:child1 -<<'EOF'
FROM example:onbuild
EOF

docker run --rm example:child1

the content of foobar is:
no-value-set-this-is-the-default

But you can set a default in the child image:

docker build --no-cache -t example:child2 -<<'EOF'
ARG REPO_DEFAULT=hey-I-am-the-child-default
FROM example:onbuild
EOF

docker run --rm example:child2

the content of foobar is:
hey-I-am-the-child-default

You can still override it at build-time, either by setting REPO_PATH as build-arg:

docker build --no-cache -t example:child3 --build-arg REPO_PATH=from-REPO_PATH-arg -<<'EOF'
ARG REPO_DEFAULT=hey-I-am-the-child-default
FROM example:onbuild
EOF

docker run --rm example:child3

the content of foobar is:
from-REPO_PATH-arg

Or REPO_DEFAULT:

docker build --no-cache -t example:child4 --build-arg REPO_DEFAULT=from-REPO_DEFAULT-arg -<<'EOF'
ARG REPO_DEFAULT=hey-I-am-the-child-default
FROM example:onbuild
EOF

docker run --rm example:child4

the content of foobar is:
from-REPO_DEFAULT-arg

And if you want to use environment-variables in your shell, you can even use those as value for the build-args:

REPO_PATH=I-am-a-shell-variable-for-REPO_PATH docker build --no-cache -t example:child5 --build-arg REPO_PATH -<<'EOF'
FROM example:onbuild
EOF

docker run --rm example:child5

the content of foobar is:
I-am-a-shell-variable-for-REPO_PATH

Same for REPO_DEFAULT as environment variable;

REPO_DEFAULT=I-am-a-shell-variable-for-REPO_DEFAULT docker build --no-cache -t example:child6 --build-arg REPO_DEFAULT -<<'EOF'
FROM example:onbuild
EOF

docker run --rm example:child6

the content of foobar is:
I-am-a-shell-variable-for-REPO_DEFAULT

[mitar]

Ah, if just args would work on Docker Hub. :-(

[JeanMertz]

Thank you for the detailed explanation @thaJeztah, works beautifully.

if just args would work on Docker Hub

Not a solution to your problem on Docker Hub @mitar, but we use Google's Container Builder at our company, and it works great (including support for build args).