special DNS record for host.docker.internal + gateway.docker.internal

[0xbad0c0d3]

I was not able to find "magic" for mac and windows systems about 'host.docker.internal' and all solutions in the Net just workarounds. Here I am with my solution... Maybe it's not the best, but it works just fine.
Yes, this functionality can be implemented with dnsmasq, but not everyone can do it by themselves. That's why i find this PR useful

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "master" git@github.com:0xbad0c0d3/libnetwork.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[pbeza]

When can we expect it to be merged into master branch?

[nyetwurk]

The way docker does DNS by default is fundamentally broken. It should always use dnsmasq and the host's DNS configuration by default (and not use 8.8.8.8). Each container should (by default) resolve DNS by querying the host, which should forward the request to its own resolver, and provide resolution for all of .docker.internal, including host.docker.internal (which should be configurable to allow containers named host). All other accessible containers should also resolve in .docker.internal

[jtaylor100]

Any idea when this will be merged? 🙂

[yinxing2008]

When will this nice feature be merged?

[Akkadius]

I have been watching this one, we have Linux members on the team that have this record broken which seems bizarre that Windows and Mac would have this record and not Linux

[shahzadmasud]

Dev team is using this one, and when pushing things on production - we are in a great challenging situation. Any anticipated fix date?

[Bardioc1977]

Dev team is using this one, and when pushing things on production - we are in a great challenging situation. Any anticipated fix date?

Hello,

if you really need a feature like this before it has been merged, you can do something like this. In our Dockerfile, we override the entrypoint.sh with something like this:

#!/usr/bin/env bash
set -x
set -e

function fix_linux_internal_host() {
DOCKER_INTERNAL_HOST="host.docker.internal"
if ! grep $DOCKER_INTERNAL_HOST /etc/hosts > /dev/null ; then
DOCKER_INTERNAL_IP='/sbin/ip route | awk '/default/ { print $3 }' | awk '!seen[$0]++'
echo -e "$DOCKER_INTERNAL_IP\t$DOCKER_INTERNAL_HOST" | tee -a /etc/hosts > /dev/null
echo "Added $DOCKER_INTERNAL_HOST to hosts /etc/hosts"
fi
}

fix_linux_internal_host

/usr/local/bin/gosu jboss:jboss bash -c "/opt/jboss/wildfly/bin/standalone.sh $@"

Be aware, that you need to do this as root user, so the change to a specific user needs to be done within entrypoint.sh (in our Wildfly version via gosu).
Afterwards, you have a "host.docker.internal" available within your container.

[nyetwurk]

Again, that workaround is known, but not generally practical for complex deployments

[Bardioc1977]

Again, that workaround is known, but not generally practical for complex deployments

I know that, thats why its called "workaround". But until the final solution is merged and available, it IS a possibility.

[schildbach]

General advice: if you restrict your PRs to the absolute bare minimum of changes necessary, you increase the chance of being merged. So never reformat any (otherwise untouched) code.

[schildbach]

The additional space is an unrelated change.

[0xbad0c0d3]

This space was produced by gofmt, are you sure it should be as you said?

[schildbach]

It doesn't matter who or what produced it. It's a change that doesn't belong to the title of this PR. As such, it will distract any reviewers of this PR.

[0xbad0c0d3]

If i'll remove it -

🐳 vet
🐳 ineffassign

🐳 fmt
sandbox.go
👹 please format Go code with 'gofmt -s -w'
Makefile:155: recipe for target 'fmt' failed
make: *** [fmt] Error 1
Makefile:114: recipe for target 'check' failed
make: *** [check] Error 2
Exited with code 2

[stativ]

Another advice, though I don't know whether this applies to docker specifically, but you usually update the pull request, rather than adding more commits that fix the problem in the original pull request.
The reason is to keep the repository history clean and concise.
What I mean, it's better to have a commit that implements the desired functionality (or a logical part of it) without any unrelated changes such as code reformatting, and then another commit that does the reformatting. Rather than having a commit that implements the functionality including unrelated changes, than a commit that reverts the unrelated changes, and finally a commit that adds them back like here.

But to end on a more positive note - I really hope this will get merged soon (whatever version will be accepted by the devs). It's an incredibly useful functionality.

[0xbad0c0d3]

Totally agree with you about the history and will keep in mind for future
;)

[0xbad0c0d3]

@schildbach, done

[Akkadius]

Let’s merge that bad boy! 😛

[simpslandyy]

Any day now for this merge! :D

[thaJeztah]

/cc @arkodg

[mavenugo]

@0xbad0c0d3 Sorry about the delayed response. couple of questions/comments :

1. I think this change will not have any effect for the default bridge (docker0) network since it doesn't use the built-in SD resolver. If we need this behavior for all the networks, then we will unfortunately have to tamper with the /etc/hosts of the container in the docker0 bridge networks.
2. What are the expectations of this fix in d4mac and d4win ? This will ofcourse return the gateway ip of the bridge which is local to the VM hosting the docker daemon. But wont directly point to the mac or win host. Will it satisfy the requirement ?

[arkodg]

@0xbad0c0d3 have not tried this out but AFAIK this will break Docker Desktop (Mac and Windows) . Today this works on Desktop through a magic proxy called vpnkit running on OSX host ( ps -e | grep com.docker.vpnkit ) for more info which allows you a way to reach the OSX host via host.docker.internal (the DNS proxy in vpnkit sends this A record to the VM)
By adding a SRV Record in the DNS Server in the dockerd daemon that maps host.docker.internal to the Gateway-IP, containers running in a native Linux environment will be able to reach the Linux Host , but on MAC, containers will be sending traffic to the Linux VM and not the OSX host

If users were able to insert A records into dockerd's DNS server through cmdline/config options and so custom mappings such as host.docker.internal could be mapped to the Host IP / or interface, this functionality could be achieved (Note: this would not work for default docker0 bridge, since for this case, the host's resolv.conf is mounted so the nameserver does not point to dockerd). I will bring up this feature with the team .

[0xbad0c0d3]

@0xbad0c0d3 have not tried this out but AFAIK this will break Docker Desktop (Mac and Windows) . Today this works on Desktop through a magic proxy called vpnkit running on OSX host ( ps -e | grep com.docker.vpnkit ) for more info which allows you a way to reach the OSX host via host.docker.internal (the DNS proxy in vpnkit sends this A record to the VM)
By adding a SRV Record in the DNS Server in the dockerd daemon that maps host.docker.internal to the Gateway-IP, containers running in a native Linux environment will be able to reach the Linux Host , but on MAC, containers will be sending traffic to the Linux VM and not the OSX host

If users were able to insert A records into dockerd's DNS server through cmdline/config options and so custom mappings such as host.docker.internal could be mapped to the Host IP / or interface, this functionality could be achieved (Note: this would not work for default docker0 bridge, since for this case, the host's resolv.conf is mounted so the nameserver does not point to dockerd). I will bring up this feature with the team .

I was thinking about the possibility to manage this feature with cmdline option, but, I am new in go, and do not have time to implement it, more because i could not test it on WIN/MAC systems... But, i think it's easy to detect OS family and apply my "fix" for linux only. Also, i realized that it would not work with docker0 because of the resolv.conf.

[arkodg]

@0xbad0c0d3 dockerd for both the scenarios (Mac and Linux) is running on Linux 😄 , for the Mac case it happens to be a VM .
https://docs.docker.com/engine/reference/commandline/run/#add-entries-to-container-hosts-file---add-host documents how you can use the --add-host argument to add entries to your container's /etc/hosts so something like docker run --add-host=host.docker.internal:172.17.0.1 --rm -it alpine ping host.docker.internal should help you reach the Linux host, but please do not add this argument while running containers on Mac or Windows (https://docs.docker.com/docker-for-mac/networking/)

[0xbad0c0d3]

@0xbad0c0d3 dockerd for both the scenarios (Mac and Linux) is running on Linux , for the Mac case it happens to be a VM .
https://docs.docker.com/engine/reference/commandline/run/#add-entries-to-container-hosts-file---add-host documents how you can use the --add-host argument to add entries to your container's /etc/hosts so something like docker run --add-host=host.docker.internal:172.17.0.1 --rm -it alpine ping host.docker.internal should help you reach the Linux host, but please do not add this argument while running containers on Mac or Windows (https://docs.docker.com/docker-for-mac/networking/)

--add-host is requiring extra work, you have to configure interface first (add alias IP), then modify all your containers to be run with --add-host, or install dnsmasq, configure it...rather than just enjoying the feature "out of the box".

[0xbad0c0d3]

@0xbad0c0d3 want to reiterate that this code will break Docker Desktop on Mac and Windows, what is needed is

1. a dockerd configuration flag called --dns-resolve-docker-host or something similar (such as --iptables) which needs to be added to https://github.com/moby/moby and enabled by default
2. Based on that flag we can programtically add a DNS entry (either in moby/moby or libnetowrk) that maps host.docker.internal to docker0's IP
3. Docker Desktop internally can then override this flag when creating a VM and spawning dockerd so that all these DNS requests go to its special proxy outside the VM

cc: @thaJeztah @justincormack @djs55

@arkodg I've implemented trough dns-opt, to switch the feature on --dns-opt resolve-docker-internal should be passed to dockerd command

[arkodg]

@0xbad0c0d3 according to https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/
--dns-opt is specific to resolve.conf, having a new flag in moby is the way to go IMHO

[jwundrak]

I can't understand, why this request is not merged. For us in an agency, we need a workaround for linux. Otherwise for WIndows or OSX, but this is the regular thing.

There are round abound more than 400 "likes" for this feature and there is nothing technical, why this can not merged.

So please merge this request.

[0xbad0c0d3]

@0xbad0c0d3 according to https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/
--dns-opt is specific to resolve.conf, having a new flag in moby is the way to go IMHO

@arkodg Generally - you are right, but in other way it requires another PR directly to docker engine, or I am wrong?

[arkodg]

@0xbad0c0d3 I think this issue can be solved completely in moby with --dns-resolve-docker-host and ExtraHosts. I can look into this next week

[arkodg]

@0xbad0c0d3 thanks for highlighting this issue, I've raised a similar PR in moby moby/moby#40007 that solves this issue using the the below approach

1. a dockerd configuration flag called --dns-resolve-docker-host or something similar (such as --iptables) which needs to be added to https://github.com/moby/moby and enabled by default
2. Based on that flag we can programtically add a DNS entry (either in moby/moby or libnetowrk) that maps host.docker.internal to docker0's IP
3. Docker Desktop internally can then override this flag when creating a VM and spawning dockerd so that all these DNS requests go to its special proxy outside the VM

[Vistrus]

Any idea when this will be merged?

[j0eii]

need this, please add it

[nazeefahmadmeer]

MERGE MERGE MERGE!

[shashank-sharma]

Looks like it will take another 3-4 months to actually solve all the dependency problems

[Akkadius]

FYI for people who have been watching this like a hawk, dialogue has been going on here between members of the Moby team: moby/moby#40007

[yilativs]

is this scheduled?

[arkodg]

@0xbad0c0d3 can you please close this PR in favor of moby/moby#40007

[omoustaouda]

In brief:

for who came here looking for how to be able to use host.docker.internal on Linux:

this is needed in your docker-compose:

in the service definition:

    name: mycontainer

   ...

    working_dir: /webhome/lumen/api
    ports:
      - "3002:3002"
      
    ...


    extra_hosts:     <---- this new part is needed
      - "host.docker.internal:host-gateway"

Then, from your container your should be able to ping the host.

$ ping host.docker.internal
PING host.docker.internal (172.17.0.1) 56(84) bytes of data.
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=2 ttl=64 time=0.096 ms
...

This should be working from Docker v. 20.10.

Thanks to everyone who worked on fixing it and who researched for solutions! 🙏☀️❤️