1.9: Default docker0 IP is not 172.17.42.1 for new installs

[ibuildthecloud]

I've gone back and forth on this one, but I do think that Docker should stick with the default docker0 IP as 172.17.42.1. On the one hand plenty will say that nobody should have never hardcoded that IP, but then on the other hand, you know people probably have.

Issues will most likely come in after 1.9 is released where the root cause will be discovered to be that the user assumed 172.17.42.1 somewhere. While the user would be at fault there is still two down sides to this. 1) The time it takes to support these issues. 2) the experience and frustration of the user that something in Docker changed and now X doesn't work.

I don't really think it's worth it to stick to the ideals that nobody should hard code this IP.

Okay, I gave my two cents.

[vdemeester]

ping @mavenugo @tiborvass

[mavenugo]

Couple of things to clarify. Any existing running docker system upgraded to 1.9.0 will retain whatever existing docker0 ip is. So, if it is 172.17.42.1 it will be just that. This issue is truly applicable ONLY for the fresh installations. @ibuildthecloud can you please update the description to reflect this.

@ibuildthecloud @tiborvass we went back and forth on this one too. 172.17.42.1 is such a low-level and undocumented thing that I could not justify the reason to provide the support. Infact, it was never guaranteed to be this IP. If in a system the subnet 172.17.0.0/16 was used, the daemon used to do the right thing by choosing another subnet from its predefined list.

Its hard to argue with @ibuildthecloud 's grounded view on the reality. I do agree that if someone hardcoded with this ip, they will know when their fresh system boots up. Infact those who are married to 171.17.42.1 can use the --bip to set that and can live happily ever after.

IMO, Encouraging such behaviour is more harmful than providing a patch for this.

Just fyi, we already have a patch for this : docker/libnetwork#649 which am very reluctant to approve.

[ibuildthecloud]

@mavenugo If a machine reboots, which IP will be assigned. On most systems docker0 is created by Docker itself. How does an upgraded setup know to keep the bridge IP as 172.17.42.1 after reboot?

[mavenugo]

@ibuildthecloud if the docker0 bridge persists (which it does) upon reboot, then the docker daemon reads the bridge IP first and use it before allocating something on its own.

[ibuildthecloud]

@mavenugo That's what I don't get, docker0 doesn't persist on reboot, right? The daemon creates it on the vast majority of systems I see. How is it persisted? Does the deb/rpm's do something to create unit files for the bridge interface.

[tiborvass]

@mavenugo For the record, I confirm that after a server restart (and docker disabled in upstart), docker0 does not come up on its own, it's docker upon starting that recreates the bridge every time.

[mrjana]

@ibuildthecloud Why should docker try to persist with the same IP for docker0 when user hasn't asked anything specifically? What need does it fulfill? I am not asking this rhetorically. I would like to understand what need does it fulfill to have a sticky IP address(or what problem does it cause if it is not sticky). We would like to avoid all these magic IP address values as much as possible which is becoming a major maintenance problem for us.

BTW, if the user wants his docker0 IP to persist he can always stick in his favorite --bip value in his favorite way in his favorite distro.

[mavenugo]

@tiborvass @ibuildthecloud yes. i stand corrected on the docker0 bridge creation on machine reboot. got carried away with the daemon restart vs machine reboot.

Regardless of that, I still dont think it is a good idea to encourage this.

[tiborvass]

I suggest to fix this with a documentation update, and possibly a daemon info log that tells which ip was used.

[burke]

For what it's worth, we have 172.17.42.1 hardcoded in a few places. I'm positive we're not the only ones, simply because -- absent a container<->host API -- it's not especially trivial to negotiate with the host how to talk to it.

The easiest method, I guess, would be to take your own IP and set the last octet to 1, but that doesn't work with host networking, where 172.17.42.1 will work just fine if you're in the same net namespace as the host.

If this IP is not hardcoded anymore, it's somewhat difficult to reliably determine the host's IP address. I guess you could introspect on the net namespace, using your own IP if it's the root ns, and clobbering the last octet if it's not? Or perhaps we just have to pass in the bridge IP via an environment variable now?

Our use-case is that we run some telemetry relay services on each host machine.

[mrjana]

@burke You should be able to look up your default gateway IP inside the container and whatever IP that is, is the IP that you can use to talk to the host. Does that work?

It's lot better than hardcoding 172.17.42.1 because it just works irrespective of which is bridge is being used, which --bip value is being used etc.

Fo detecting whether you are in host net namespace you can look for the presence of docker0 bridge inside the container. If it is present then you can simply pick docker0's IP address.

So to me, I don't think we need a special container<->host api to achieve this.

[burke]

Asking for the gateway works as long as the container knows it's running in a network namespace:

$ docker run ubuntu:14.04 bash -c "ip ro get 8.8.8.8 | grep -oP '(?<=via )([\d\.]+)'"
172.17.42.1
$ docker run --net=host ubuntu:14.04 bash -c "ip ro get 8.8.8.8 | grep -oP '(?<=via )([\d\.]+)'"
172.16.4.1

The former is correct, but the latter is our network hardware. To work around this, I have to either tell my containers whether they're running with host networking, or teach them the subnets we use in our datacenters so that they can go "aha, this is DC X, not a docker virtual network, so I will use my own IP, not my gateway, to talk to the host".

To be clear, this issue isn't a huge dealbreaker for me -- I can just pass yet another environment variable into the container -- but I think it would be valuable for the containerized app to have a predictable way of routing to its host.

[cpuguy83]

@burke @ibuildthecloud But you can use --bip to ensure the same bridge net.

[burke]

Oh! I missed that. Sorry, disregard, that'll work just fine.

[mavenugo]

As discussed, docker#17384 adds the required info log to the user to notice and take action if required. (I hope there isnt anyone who relies on these magic ip-addresses)

[tiborvass]

Closed by #17384

[ibuildthecloud]

Fairwell 172.17.42.1, you will be missed.

[tiborvass]

@ibuildthecloud feel free to resuscitate it with --bip

[ibuildthecloud]

I will :)

[thaJeztah]

@tiborvass not sure where we should add this to the documentation, but would it be good to mention in the release notes?

[luebken]

@tiborvass not sure where we should add this to the documentation, but would it be good to mention in the release notes?

Well it's to late for the release notes. ;-) But a +1 for documenting the change itself somewhere. I also would like to learn more about the reasoning.

[wesleymusgrove]

@ibuildthecloud, can you point me to any documentation for how to use --bip to ensure the docker0 adapter gets assigned 172.17.42.1 every time?

Unfortunately this change bit me because I'm behind a corporate firewall and must use SSH tunneling to port forward traffic from 172.17.42.1 to a proxy. I didn't expect this to change so I hardcoded 172.17.42.1 all over the place in my project's code, environment variables, etc... Thanks!

[mavenugo]

@wesleymusgrove Pls refer to https://github.com/docker/docker/blob/e44364eae90784b423eee8b2969bda9cd2429746/man/docker-daemon.8.md#options. The documentation for the --bip option is hopefully clear. This is a docker daemon option.

[wesleymusgrove]

Thanks @mavenugo! sudo docker daemon --bip="172.17.42.1/16" temporarily fixed the issue for me. After a host restart the docker0 IP reverted back to 172.17.0.1. Is there any way to get it to persist?

[mavenugo]

@wesleymusgrove please make sure the flag --bip is used by the docker service when your host restarts. The service configuration will be different for different distros. Pls refer to https://docs.docker.com/engine/articles/configuring/ and configure the daemon appropriately.

[wesleymusgrove]

Thanks again @mavenugo! On Ubuntu 14.04 setting DOCKER_OPTS="--bip 172.17.42.1/16" in /etc/default/docker solved the issue for me.

[vmatekole]

Today one our CoreOS clusters went down in production due to this change. Consul failed to start, as a result all services were unable to be discovered. It appears this may have come about after a leader host was rebooted and switched over to a later version of docker via a CoreOS update. Docker daemon in CoreOS is normally configured via cloud-config which is immutable... The cluster had to be rebuilt with a new config forcing the gateway to be set at the original 172.17.42.1 as per above. I'm not sure how we can avoid hard coding the gateway IP in cloud config... Nonetheless, it's unfortunate this had to change to cause us a costly downtime.

[mavenugo]

@vmatekole as discussed in this thread, its a bad practice to depend on 172.17.42.1 as a constant as there is no such guarantee supported by docker even in the older versions. Our recommendation for anyone still depending on this IP is to use the --bip flag (docker#17305 (comment)).

[vmatekole]

@mavenugo I understand this is now stipulated but it appears the docker ecosystem incl. Consul and CoreOS fail to indoctrinate or evangelise this practice, at least from what I can ascertain. And I have never come across this in documentation, even the link - https://docs.docker.com/engine/admin/configuring/ you provide on this practice, appears to have the info buried in a place that I can't find.... I understand no guarantees were made but there was little warning either and as such, practices have followed suit...

[MrMMorris]

Found this issue when I realized my docker containers weren't resolving Consul dns after moving from 1.6.2 to 1.9.1. I was using --dns=172.17.42.1. With --dns=172.17.0.1 it works again.

My question is, should I set --bip 172.17.0.1/16 to be safe or will it be that forever?

[mavenugo]

@MrMMorris its not a good idea to depend on any such magic IPs. If you have to depend on such an IP, it must be specified via --bip and that will make sure the docker0 bridge will be assigned that (or) fail if it cannot.

[MrMMorris]

@mavenugo well from what I remember, the only reason I initially set --dns was because docker didn't use the hosts resolv.conf and was a bug. I would like to not have to set it, but consul dns resolution does not work inside a container if I don't

[wkruse]

As @MrMMorris I am using --dns=172.17.42.1 and as @vmatekole I've got bitten after CoreOS upgrade. The reason is a containerized DNS server (SkyDNS) and when container from the same Docker host are talking to it through Docker host IP, the DNS resolution breaks, because the DNS response arrives from docker0 (the IP in DNS request doesn't match the IP in the DNS response). I'll set --bip to fix it. But are there other ways to get containerized DNS working (besides --net=host and --dns={docker0})?

[MrMMorris]

I am not able to get consul DNS to work inside containers without DOCKER_OPTS="--bip=172.17.0.1/16 --dns=172.17.0.1 --dns=8.8.8.8". Still not sure if that's a bug or if I am missing something.

@mavenugo if you say "its not a good idea to depend on any such magic IPs", do you know why DNS inside containers does not work? I thought they just inherited the settings from the host. I am using dnsmasq and I have it set up to accept connections from docker0 with interface=docker0. Not sure what could be missing.