Propagate http_proxy env var to builds

[crosbymichael]

I'm just opening this issue from the discussion in #1852. The original issue was fixed but this is a secondary discussion about the need for this env var to be available from the host in builds so that the containers can access the internet.

From the build POV we don't want to commit this because it really makes things non-portable but maybe we can find a good solution for everyone.

[wyaeld]

mentioning interesting parties from the tail of the previous issue @dangra @codeaholics

[creack]

I think Docker should setup a userland proxy when http_proxy is set so the containers do not need to worry about it.

[codeaholics]

You still have to pass http_proxy to the build containers to get them to use the proxy, no? In which case having a user land proxy on the host is no different from having a proxy somewhere on the network and doesn't alter this issue except by adding complexity. :-)

[dangra]

I see everyone is interested in getting http proxy into the build, I was more keen to see a generic solution for environment variables available at build time and not burned into final images.

In my case I would like to set PIP_INDEX_URL pointing to an internal pypi cache that speeds up python builds, although it is an optional envvar that works for builds without it.

[tianon]

To quote myself from that previous thread:

if we push http_proxy and https_proxy down into the containers themselves, we're setting ourselves up for a repeat of #4181, where a lot of people will set http_proxy to 127.0.0.1 (which is fairly common practice, just like having a nameserver of 127.0.0.1), and then wonder why their containers decide they can't access HTTP sites even though ping works fine (because 127.0.0.1 inside the container refers to the container, not the host)

[wyaeld]

@tianon I see your point.

I've been wondering for a while why Docker doesn't expose an easy hostname to refer to the host, and avoid the problems you point out with 127.0.0.1 type values going in.

With something like that, this feature could look like:

docker build -e http_proxy=DOCKER_HOST:3128 -t wyaeld/ubuntu .

and then

docker run -i -t wyaeld/ubuntu /bin/bash

and have there be no http_proxy baked in to the image. Now its portable again.

[wyaeld]

@crosbymichael Does the new ability to use host networking solve this issue wrt proxies? I haven't looked too closely yet to see if that can be used with the builder.

[jrandall]

I've also been suffering from the issue of not being able to run docker build from behind the corporate firewall. I think the proposed solutions for passing environment into the build might work, but they seem dangerous to portability.

I've come up with an alternative solution, called docker-proxify for building container images from behind a corporate proxy. It consists of a transparent proxy server installed into a docker container which is also capable of running docker-within-docker. The result is that you can configure the proxies during normal run-time configuration of the outer container, and then as a result of the transparent proxying, no longer have to worry about proxy settings in the inner containers (including those invoked during docker build on a Dockerfile).

It's in early days, but is working right now for http and https on standard ports. Try it yourself - I'd appreciate feedback and bug reports.

The easiest way to run it is with the trusted build I've added to the index:

$ docker run -i -t --privileged -e http_proxy -e https_proxy jrandall/docker-proxify
root@cfe1d7f50ae7:/docker# docker build -q github.com/dockerfile/ubuntu
...
Successfully built afcd247466a7
root@4ad182e3a976:/docker# docker run -i -t afcd247466a7
[ root@5556a3ca0a17:~ ]$

[larrycai]

@jrandall : this looks a very interesting, though I still prefer to solve it in docker itself.

[jalev]

+1

@jrandall Your solution is brilliant and has solved me a massive headache we're having. I'm running it with --net=host and turning it into a host-wide forwarding-proxy, which solves our proxified building problem quite nicely.

I do agree with @larrycai that this should be solved by Docker in the future, but in the mean time docker-proxify is a good intermediary solution.

[mapuri]

Hello:

I am a newbie to the docker community. I faced a similar issue as described in this thread while trying to setup the docker build environment. My machine sits behind a http-proxy. And I had to modify the DockerFile in the source for make build to succeed, as it also pulls multiple artifacts from the web. While making changes to a DockerFile is easy but the changes I made are obviously not something I could contribute back. Also I think that this might be a use-case in some other environments that want to use third-party DockerFiles to build their containers but don't have control over the DockerFile source (unless they copy and maintain the file themselves).

From the thread I understand the portability concerns associated with the docker containers built using host specific environment variables, in case we add similar support as docker run -e to docker build.

In order to address the above concern I propose that we introduce a notion of build-time environment variables. A build-time environment variable shall get used only while processing the 'RUN' primitive of a DockerFile. Such a variable shall only be accessible during 'RUN' and shall not be persisted with the intermediate and/or final docker images themselves, thereby addressing the portability concerns of the images.

I have a patch that I tried out in my environment and appears to be working with some basic testing. I wanted to share it here to see if this can be a useful addition and get your feedback. I would be happy to contribute if there is enough interest.

I will create a pull request shortly, my changes are in my forked repo for your reference: https://github.com/mapuri/docker

[mapuri]

Folks, just FYI I have created a PR #9176 to introduce the build-time environment variables. Kindly let me know your feedback.

[duglin]

@crosbymichael you ok if we close this and keep the conversation going in #9176 ?

[danday74]

We are doing ...

ENV http_proxy http://9.9.9.9:9999
ENV https_proxy http://9.9.9.9:9999

and at end of dockerfile ...

ENV http_proxy ""
ENV https_proxy ""

This, for now (until docker introduces build env vars), allows the proxy vars to be used for build without publicly exposing them

[jrandall]

This issue recently came up in GNU guix, which also uses a form of containerisation during the build process. The solution there was to create a core configuration option called leaked_env_vars (https://www.gnu.org/software/guix/manual/html_node/Derivations.html#index-derivation-1) which defines a list of environment variables that are injected into the build environment from the outer environment. I'd be supportive of a similar option for docker (the "outer environment" could be that of the docker daemon or of the docker build client, whichever is easier) - I think this is basically Issue #9176.

[jrandall]

@danday74 that works for sharing the image but not the Dockerfile itself (and critically, not for building a Dockerfile from elsewhere that doesn't include proxy vars that match your environment).

[thaJeztah]

Duplicated by #14634?

[thaJeztah]

There's now a new issue created covering this so I'll close this issue.

For those participating / following this, please continue the discussion in #14634