New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for environment variables for building containers #6822

Closed
smysnk opened this Issue Jul 2, 2014 · 25 comments

Comments

Projects
None yet
@smysnk

smysnk commented Jul 2, 2014

It would be nice to be able to use --env to specify environment variables that can be used in the docker build process. Currently environment variables are only supported as part of the docker run command.

An example use case for this scenario:

I use scp to copy files into my container as to avoid creation of unnecessary layers using the ADD / RUN commands. Sometimes I will change the DOCKER_HOST depending on docker networking setup (docker in docker sometimes uses 10.x.x.x range).

$ docker build --env DOCKER_HOST=172.17.42.1 -t foo .

Dockerfile
FROM private/base
RUN scp -r vagrant@$DOCKER_HOST:/vagrant/docker/foo/* . && /bin/bash /fs/build.sh
CMD ["/bin/bash", "/bootstrap.sh"]

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 2, 2014

Contributor

This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds.

This particular use-case would be better served by being able to remove or squash layers that contain build artifacts which aren't required for the final image. Such as #4232

Contributor

cpuguy83 commented Jul 2, 2014

This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds.

This particular use-case would be better served by being able to remove or squash layers that contain build artifacts which aren't required for the final image. Such as #4232

@cpuguy83 cpuguy83 closed this Jul 2, 2014

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Jul 2, 2014

Collaborator

@smysnk with the 1.1.0 release (coming very soon), you'll be able to have a builder image and do something like docker run -e FOO=bar builder_image | docker build - provided the builder_image when run sends out an image's tar (docker save).

Collaborator

tiborvass commented Jul 2, 2014

@smysnk with the 1.1.0 release (coming very soon), you'll be able to have a builder image and do something like docker run -e FOO=bar builder_image | docker build - provided the builder_image when run sends out an image's tar (docker save).

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 2, 2014

Contributor

Woot, cool

Contributor

cpuguy83 commented Jul 2, 2014

Woot, cool

@josh-devops-center

This comment has been minimized.

Show comment
Hide comment
@josh-devops-center

josh-devops-center Dec 28, 2014

Docker supports the ONBUILD instruction which allows for a delayed instruction to run. This allows for a template Dockerfile to be inherited and extended.

@cpuguy83 Why can't a similar mechanism be done? You say the following, though where is the discussion and what's wrong with doing something similar to ONBUILD? It's not clear why the "ONLY" option is to use env in the build command, rather than supporting this in the Dockerfile itself.

"This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds."

josh-devops-center commented Dec 28, 2014

Docker supports the ONBUILD instruction which allows for a delayed instruction to run. This allows for a template Dockerfile to be inherited and extended.

@cpuguy83 Why can't a similar mechanism be done? You say the following, though where is the discussion and what's wrong with doing something similar to ONBUILD? It's not clear why the "ONLY" option is to use env in the build command, rather than supporting this in the Dockerfile itself.

"This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds."
@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 30, 2014

Contributor

@josh-devops-center I'm not sure I follow how else this would be supported in the Dockerfile.

I should say, I'm not necessarily against having support for envs provided to the build command, and also IANTM, just relaying from many past conversations around this topic.

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

Contributor

cpuguy83 commented Dec 30, 2014

@josh-devops-center I'm not sure I follow how else this would be supported in the Dockerfile.

I should say, I'm not necessarily against having support for envs provided to the build command, and also IANTM, just relaying from many past conversations around this topic.

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

@brettviren

This comment has been minimized.

Show comment
Hide comment
@brettviren

brettviren Dec 30, 2014

One use case to support this feature: I'm trying to use Docker to provide "standard" platforms for building our software in such a way that they can be exercised portably. However, site-specific things have to be worked around. In particular, setting the various "http_proxy" variables are needed on some but not all build hosts. A secondary use is to offer the resulting images for download and these should not ultimately bake-in the build host's HTTP proxy variables.

With "docker build --env-file=site.env [...]" I would be able to deal with the first use case easily by maintaining a build-host-specific site.env file. I'd still need a way to null these out (as a final build command) for the final image to support the second use case.

The only good alternative to adding --env-file/--env to "docker build" that I can see is to maintain my Dockerfiles with some templating system so that they themselves are built from a set of site-neutral files + site-specific definitions.

Thanks for any considerations on this!

brettviren commented Dec 30, 2014

One use case to support this feature: I'm trying to use Docker to provide "standard" platforms for building our software in such a way that they can be exercised portably. However, site-specific things have to be worked around. In particular, setting the various "http_proxy" variables are needed on some but not all build hosts. A secondary use is to offer the resulting images for download and these should not ultimately bake-in the build host's HTTP proxy variables.

With "docker build --env-file=site.env [...]" I would be able to deal with the first use case easily by maintaining a build-host-specific site.env file. I'd still need a way to null these out (as a final build command) for the final image to support the second use case.

The only good alternative to adding --env-file/--env to "docker build" that I can see is to maintain my Dockerfiles with some templating system so that they themselves are built from a set of site-neutral files + site-specific definitions.

Thanks for any considerations on this!

@josh-devops-center

This comment has been minimized.

Show comment
Hide comment
@josh-devops-center

josh-devops-center Dec 30, 2014

@cpuguy83

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

This works. The ability to specify an environment file that the build then sources so that the environment variables are effective for all subsequent works nicely. We could then have a generic Dockerfile that acts as the template, and then rely on the local context for environmental builds which are repeatable. This capability be great to have in Docker.

josh-devops-center commented Dec 30, 2014

@cpuguy83

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

This works. The ability to specify an environment file that the build then sources so that the environment variables are effective for all subsequent works nicely. We could then have a generic Dockerfile that acts as the template, and then rely on the local context for environmental builds which are repeatable. This capability be great to have in Docker.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 30, 2014

Contributor

@josh-devops-center I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

Contributor

cpuguy83 commented Dec 30, 2014

@josh-devops-center I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

@josh-devops-center

This comment has been minimized.

Show comment
Hide comment
@josh-devops-center

josh-devops-center Dec 30, 2014

@cpuguy83

 I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

How would this work for a template style approach as both myself and @brettviren asked for? I'm not looking to build the image until the environmental variables are set. In fact, in certain environment specific builds the image will not build successfully until the environment variables are all properly set.

I'm looking for a way to delay setting the environment variables, similar to the effect that ONBUILD has. This would then allow for a templated Docker build for different environments.

josh-devops-center commented Dec 30, 2014

@cpuguy83

 I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

How would this work for a template style approach as both myself and @brettviren asked for? I'm not looking to build the image until the environmental variables are set. In fact, in certain environment specific builds the image will not build successfully until the environment variables are all properly set.

I'm looking for a way to delay setting the environment variables, similar to the effect that ONBUILD has. This would then allow for a templated Docker build for different environments.

@brettviren

This comment has been minimized.

Show comment
Hide comment
@brettviren

brettviren Dec 30, 2014

FYI, today I played around with this. I'm still learning Docker so maybe what I'm doing is "bad". It's here: https://github.com/LBNE/lbne-docker

In short: I "ADD" dot.profile -> .bash_profile file and have this set any proxy env variables depending on what network it is on based on what is found in /etc/resolv.conf which itself is set by the host admin. Then, any commands that may need to download through our proxy get "RUN" with "bash -l -c '...'" which implicitly sources the .bash_profile and assures ftp_proxy, etc is set. It feels pretty kludgey, but so far it is working. Something cleaner would be welcome!

Also, as you may also notice, instead of templates I used a simple config file + docker-map/docker-py to generate the Dockerfile.

brettviren commented Dec 30, 2014

FYI, today I played around with this. I'm still learning Docker so maybe what I'm doing is "bad". It's here: https://github.com/LBNE/lbne-docker

In short: I "ADD" dot.profile -> .bash_profile file and have this set any proxy env variables depending on what network it is on based on what is found in /etc/resolv.conf which itself is set by the host admin. Then, any commands that may need to download through our proxy get "RUN" with "bash -l -c '...'" which implicitly sources the .bash_profile and assures ftp_proxy, etc is set. It feels pretty kludgey, but so far it is working. Something cleaner would be welcome!

Also, as you may also notice, instead of templates I used a simple config file + docker-map/docker-py to generate the Dockerfile.

@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Feb 20, 2015

@josh-devops-center Did you get anywhere with your ONBUILD environment variables idea? I also think that it could be a reasonable way on controlling the build.

charandas commented Feb 20, 2015

@josh-devops-center Did you get anywhere with your ONBUILD environment variables idea? I also think that it could be a reasonable way on controlling the build.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 21, 2015

Member

For those following this issue; there's a PR for supporting build-time environment variables here; #9176

Member

thaJeztah commented Feb 21, 2015

For those following this issue; there's a PR for supporting build-time environment variables here; #9176

@freeformz

This comment has been minimized.

Show comment
Hide comment
@freeformz

freeformz Aug 18, 2015

It looks like #14634 brings this up again.

freeformz commented Aug 18, 2015

It looks like #14634 brings this up again.

@skodumuru

This comment has been minimized.

Show comment
Hide comment
@skodumuru

skodumuru Dec 30, 2015

i have a Dockerfile in which i am running wget command as RUN wget --user $USER --password $PASSWORD ...... I am using docker-compose up command to make dockers up. Before this command i am setting as export $USER=skodumuru and same for password also. But this is not working. Is there any way of doing this without using shell script file?

skodumuru commented Dec 30, 2015

i have a Dockerfile in which i am running wget command as RUN wget --user $USER --password $PASSWORD ...... I am using docker-compose up command to make dockers up. Before this command i am setting as export $USER=skodumuru and same for password also. But this is not working. Is there any way of doing this without using shell script file?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 30, 2015

Member

@skodumuru you can use the build-time variables, that were implemented in #15182. HOWEVER, keep in mind that passing credentials like that is insecure, and the credentials will be baked into the resulting image (or its "history"), so you don't want to share that image with other people if you don't want them to know your username/password

Member

thaJeztah commented Dec 30, 2015

@skodumuru you can use the build-time variables, that were implemented in #15182. HOWEVER, keep in mind that passing credentials like that is insecure, and the credentials will be baked into the resulting image (or its "history"), so you don't want to share that image with other people if you don't want them to know your username/password

@skodumuru

This comment has been minimized.

Show comment
Hide comment
@skodumuru

skodumuru Dec 30, 2015

seems that build environment variables are for docker build command but we are using docker-compose command. how can i pass ENV variables with this command.

skodumuru commented Dec 30, 2015

seems that build environment variables are for docker build command but we are using docker-compose command. how can i pass ENV variables with this command.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 30, 2015

Member

@skodumuru I don't think build-args are supported yet on compose, but here's an issue tracking that; docker/compose#2111

Member

thaJeztah commented Dec 30, 2015

@skodumuru I don't think build-args are supported yet on compose, but here's an issue tracking that; docker/compose#2111

@Jason-Grant

This comment has been minimized.

Show comment
Hide comment
@Jason-Grant

Jason-Grant Dec 31, 2015

I find a way to build containers depend on environment variables (actually they are build arguments), but work like environment variables. The bad thing about this method is that you can not use it when you build your containers by docker-compose, like @thaJeztah said build-args are not supported yet on docker-compose.

Here is the method

FROM php:7.0-fpm
ARG APP_ENV=local
ENV APP_ENV ${APP_ENV}
RUN cd /usr/local/etc/php && ln -sf php.ini-${APP_ENV} php.ini

you can run docker build --build-arg APP_ENV=prod . to build a production container.

Jason-Grant commented Dec 31, 2015

I find a way to build containers depend on environment variables (actually they are build arguments), but work like environment variables. The bad thing about this method is that you can not use it when you build your containers by docker-compose, like @thaJeztah said build-args are not supported yet on docker-compose.

Here is the method

FROM php:7.0-fpm
ARG APP_ENV=local
ENV APP_ENV ${APP_ENV}
RUN cd /usr/local/etc/php && ln -sf php.ini-${APP_ENV} php.ini

you can run docker build --build-arg APP_ENV=prod . to build a production container.

@lrepolho

This comment has been minimized.

Show comment
Hide comment
@lrepolho

lrepolho Aug 19, 2016

Is there anyway I can replace FROM instruction with a --build-arg. ?

Example:

$ docker build --build-args region=us-east-1 .
// Dockerfile
FROM aws.ecr.huge.url.${region}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"

lrepolho commented Aug 19, 2016

Is there anyway I can replace FROM instruction with a --build-arg. ?

Example:

$ docker build --build-args region=us-east-1 .
// Dockerfile
FROM aws.ecr.huge.url.${region}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"
@marcbachmann

This comment has been minimized.

Show comment
Hide comment
@marcbachmann

marcbachmann Aug 21, 2016

Is there anyway I can replace FROM instruction with a --build-arg. ?

I'm currently using sed for this:

cat Dockerfile | sed "s/{{region}}/us-east-1/" | docker build -t php-apache -

with a Dockerfile that looks like that:

from aws.ecr.huge.url.{{region}}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"

marcbachmann commented Aug 21, 2016

Is there anyway I can replace FROM instruction with a --build-arg. ?

I'm currently using sed for this:

cat Dockerfile | sed "s/{{region}}/us-east-1/" | docker build -t php-apache -

with a Dockerfile that looks like that:

from aws.ecr.huge.url.{{region}}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"
@Alexhha

This comment has been minimized.

Show comment
Hide comment
@Alexhha

Alexhha Apr 11, 2017

Just alternative solution - using envsubst command from moreutils package

$ DOCKER_HOST=localhost envsubst < Dockerfile
FROM private/base

RUN scp -r vagrant@localhost:/vagrant/docker/foo/* . \
    && /bin/bash /fs/build.sh

CMD ["/bin/bash", "/bootstrap.sh"]

So the final command is

$ DOCKER_HOST=localhost envsubst < Dockerfile | docker build -t php-apache -

You can pass any number of variables

$ PARAM1=VAL1 PARAM2=VAL2 ... envsubst < Dockerfile | docker build -t php-apache -

Alexhha commented Apr 11, 2017

Just alternative solution - using envsubst command from moreutils package

$ DOCKER_HOST=localhost envsubst < Dockerfile
FROM private/base

RUN scp -r vagrant@localhost:/vagrant/docker/foo/* . \
    && /bin/bash /fs/build.sh

CMD ["/bin/bash", "/bootstrap.sh"]

So the final command is

$ DOCKER_HOST=localhost envsubst < Dockerfile | docker build -t php-apache -

You can pass any number of variables

$ PARAM1=VAL1 PARAM2=VAL2 ... envsubst < Dockerfile | docker build -t php-apache -
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 11, 2017

Member

Docker 17.05 adds support for using ARG in FROM, so;

ARG MYTAG=latest
FROM foobar:$MYTAG

See this PR; #31352

Member

thaJeztah commented Apr 11, 2017

Docker 17.05 adds support for using ARG in FROM, so;

ARG MYTAG=latest
FROM foobar:$MYTAG

See this PR; #31352

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Apr 11, 2017

Contributor

Is there anyway I can replace FROM instruction with a --build-arg

Yes, coming up in 17.05

Contributor

cpuguy83 commented Apr 11, 2017

Is there anyway I can replace FROM instruction with a --build-arg

Yes, coming up in 17.05

@hatdropper1977

This comment has been minimized.

Show comment
Hide comment
@hatdropper1977

hatdropper1977 Jul 20, 2017

@thaJeztah I tried your syntax and get an error -- Compose wants the FROM to precede the ARG. For example, in my Dockerfile I have:

 ARG es_version
 FROM docker.elastic.co/elasticsearch/elasticsearch:$es_version

I get this error:

 Building elasticsearch
 Step 1/14 : ARG es_version
 ERROR: Service 'elasticsearch' failed to build: Please provide a source image with `from` prior to commit

hatdropper1977 commented Jul 20, 2017

@thaJeztah I tried your syntax and get an error -- Compose wants the FROM to precede the ARG. For example, in my Dockerfile I have:

 ARG es_version
 FROM docker.elastic.co/elasticsearch/elasticsearch:$es_version

I get this error:

 Building elasticsearch
 Step 1/14 : ARG es_version
 ERROR: Service 'elasticsearch' failed to build: Please provide a source image with `from` prior to commit
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 20, 2017

Member

@hatdropper1977 what version of Docker are you running, and what version of docker compose? Does it work correctly if you build using docker build?

Member

thaJeztah commented Jul 20, 2017

@hatdropper1977 what version of Docker are you running, and what version of docker compose? Does it work correctly if you build using docker build?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment