Public IP issues in docker network

[geekthattweaks]

Description of problem:
We are encountering an unexpected networking disruption with the new docker networking added in 1.10.0.
Our setup includes adding multiple public IP ranges to a single device. Whilst any IP in a range is bound to an interface on the host, the container using the specific network for that range will have no network connectivity. Only once the IP is removed from the host interface will the container regain network connectivity.
Once then initially resolved, the range operates as expected for any existing or new containers without further problem. Our specific use case and problem is that the first public range also includes our “primary” ip to provide SSH and cannot be removed without losing remote access.

docker version:
Client:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d510
Built: Thu Feb 4 18:36:33 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d510
Built: Thu Feb 4 18:36:33 2016
OS/Arch: linux/amd64

docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 2
Server Version: 1.10.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 14
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.38 GiB
Name: HOSTNAME
ID: KLDH:Z2IC:SLXE:M47A:PSBN:AO7W:NPC4:XHIT:ZSGH:YJY7:VJGB:3N3U
Username: USERNAME
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

uname -a:
Linux HOSTNAME 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
Physical server

How reproducible:
Simple to reproduce, but requires multiple public IP ranges to test.

Steps to Reproduce:

1. Add the first usable ip address from a public range to interface. In our test we use ip addr add. ip addr add xxx.xxx.xxx.122/29 dev eth0
2. Create a docker network with the range from step 1. docker network create --subnet=xxx.xxx.xxx.120/29 --gateway=xxx.xxx.xxx.121 example
3. Create container using a different ip from step 1 but in the same range. docker --net=example --ip=xxx.xxx.xxx.123 ubuntu ping google.com
4. Check output from container, pings should fail: “Destination Host Unreachable”

-- the following steps are a quick remedy for the issue, but steps to reproduce end here --

1. Remove the IP address from step 1 from interface ip addr del xxx.xxx.xxx.122/29 dev eth0
2. Check output from container; pings should start operating normally within a few seconds.

Actual Results:
Any containers created with the specified network fail to gain network connectivity.

Expected Results:
The container should have connectivity even if the host has an IP in the range bound to the network interfaces.

Additional info:
Thank you having looking into this and please let me know if you need any further info!

[aboch]

@geekthattweaks
Docker manages the networking (IPAM and Network plumbing) for the networks (and relative containers) it creates. It is not responsible of the other interfaces in your host.

In your use-case, having two interfaces (eth0 and bridge interface) assigned addresses in the same subnet, may require extra manual configurations so that Linux properly forwards packets to/from the bridge interface.

I will look at some options to get this working for you, but I just want to clarify that as I understand it, this is not a Docker networking issue, therefore please be aware responses to this issue may take some time.

Thanks.

[tiborvass]

Per @aboch's comment, I'm closing this. Discussions can continue. Feel free to speak up if you think this is a docker issue.

[CaptainYarb]

@aboch Thanks for the update. I'm also watching this issue and I'm not entirely certain that I agree with your perspective on this.

While it's true that Docker shouldn't have to manage the other interfaces, I'm not certain that I agree that ignoring them is a better option. It's fairly typical that most physical server providers allocate /29's to the servers themselves. Without the ability to handle this use case, the docker networking tool will be completely useless to provide the feature-set for 4-5 IP addresses.

The more interesting part of this is the fact that we were able to effectively setup the IP -> Container routing before the RC builds or 1.0.10 builds by just using bridges and iptables and didn't encounter a similar problem. There are a few minor differences in how we handled the bridges, but the iptables rules were effectively the same thing.

I look forward to seeing more details regarding this issue. Thanks for your time!