Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker gives out IP addresses already in use in bridge mode #11199

Closed
Johnsmith1111 opened this issue Mar 6, 2015 · 25 comments

Comments

@Johnsmith1111
Copy link

commented Mar 6, 2015

/system/networking,Please.
So,I feel badly today,When I use docker in the test environment in my company with bridge mode which the contaner use the same ip block with host.when I start a container,after a few minute

ONE OF OUR SYSTEM IS DOWN BECAUSE OF IP ADDRESS CONFLICT

AND THE CONFLICT IP IS THE CONTAIN'S IP

And I'm headache about the docker's network for a long time.

Why docker manage IP address for itself?

And why IP CONFLICT could happened with docker?

I feel so badly today.

@moypray

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2015

You can restart your docker daemon with --bip parameter to set your docker bridge IP.
The container IP will be in the same net address. [ I am sure it you will not conflict again.]

  1. if docker0 is exist, you must delete docker0 first manually.
  2. docker -d --bip=1.1.1.1/16 xxxx
    ....
    Containers will allocate IP in net 1.1.x.x.
@Johnsmith1111

This comment has been minimized.

Copy link
Author

commented Mar 6, 2015

@moypray I just want to know why docker make this low-level mistake without any document info.
And I make a bridge to eth0.So the container could communicate with ohter server just as vm.
And..every day I use docker I feel just as fucking a dog

@estesp

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2015

Can you help us by explaining a bit more what you mean by: "with bridge mode which the container use the same ip block with host"? Docker tries very hard to pick an IP address range for the containers which is not found in any current routes on any interface on the host. You can check this code for yourself in https://github.com/docker/docker/blob/master/daemon/networkdriver/bridge/driver.go#L375-L387

Also in that file are a long list of private networks that it will attempt in the "range addrs" loop I linked to above. The checking code actually uses netlink to list all routes (see the first column of ip route) and checks nameservers for network overlap as well.

The only way I see Docker doing this "on purpose", is if you created the bridge yourself before starting Docker and already gave it an IP address which allowed it to hand out IPs from your specified CIDR.

If you didn't do that, could you please provide the following details so we can look into the problem further and figure out if there is a bug in this checking code:

  1. Provide docker version and docker info output
  2. Output of your bridge configuration as it is now (ip -4 addr show dev {bridgeDevice})
  3. ip route output on your host
  4. The flags being passed to your docker daemon invocation (if you don't know, you can ps axww | grep docker and it should show you the current flags if it is running)
@Johnsmith1111

This comment has been minimized.

Copy link
Author

commented Mar 7, 2015

@estesp
Yes,I create bridge to eth0 by myself,It's handy for manage contaner if they can
accessed by ip,Any question?

2015-03-06 22:42 GMT+08:00 Phil Estes notifications@github.com:

Can you help us by explaining a bit more what you mean by: "with bridge
mode which the container use the same ip block with host
"? Docker tries
very hard to pick an IP address range for the containers which is not found
in any current routes on any interface on the host. You can check this code
for yourself in
https://github.com/docker/docker/blob/master/daemon/networkdriver/bridge/driver.go#L375-L387

Also in that file are a long list of private networks that it will attempt
in the "range addrs" loop I linked to above. The checking code actually
uses netlink to list all routes (see the first column of ip route) and
checks nameservers for network overlap as well.

The only way I see Docker doing this "on purpose", is if you created the
bridge yourself before starting Docker and already gave it an IP address
which allowed it to hand out IPs from your specified CIDR.

If you didn't do that, could you please provide the following details so
we can look into the problem further and figure out if there is a bug in
this checking code:

  1. Provide docker version and docker info output
  2. Output of your bridge configuration as it is now (ip -4 addr show
    dev {bridgeDevice})
  3. ip route output on your host
  4. The flags being passed to your docker daemon invocation (if you
    don't know, you can ps axww | grep docker and it should show you the
    current flags if it is running)


Reply to this email directly or view it on GitHub
#11199 (comment).

@programmerq

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2015

I remember running into this same concern when I was new to docker!

There is a bit of a case of information overload where this is found in the docs, and the consequence you had happen isn't immediately visible without some testing: http://docs.docker.com/articles/networking/#customizing-docker0

To get in the situation you described, I believe you would have to be using the --bip flag as an argument when you run the docker daemon. The --bip flag tags in a range of IP addresses in CIDR format. You are effectively are telling docker "allocate IP addresses in this range for containers" with this option.

After looking over the docs, it does look like it might be handy to make this a little more obvious. I can try to work up a patch to the docs to help make this more clear. @shi154462671 Do you have any thoughts on smoothing out any trouble spots you ran into in the docs?

@Johnsmith1111

This comment has been minimized.

Copy link
Author

commented Mar 7, 2015

@programmerq
I create a bridge to eth0.NOT THE DEFAULT DOCKER0
and get in the situation is easy.
just create two vm,and configure each with birdge to eth0.
and the argument is -b=mybridgetoeth0
NOT bip0
Then run a contain in each vm,
AND YOU CAN SEE THEY USE THE SAME IP.
And the doc you refer I already read it for many times.

@estesp

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2015

That's an interesting configuration, but as you can understand, bridging the same exact interface in two different daemons will result in both thinking they are operating on the network CIDR of the bridge interface, which probably in your case meant they both assumed they were handed the network range associated with eth0 that you bridged to.

Given there is nothing in the docs that would recommend such a configuration (and the docs clearly state it will use the CIDR of the bridge as the address range for container IP assignment), what changes would you recommend to Docker to help you and others who may try what you tried in the future? It should be clear that Docker did not do this on its own, and instead will use private address ranges that do not overlap with any other address ranges that it can find on the system. Allowing users to configure things exactly as they want means users also have the power to shoot themselves in the foot at times. But, if there is anything that Docker can improve in docs or implementation, we're all ears to try and make things better within feasibility and time constraints, of course :)

@Johnsmith1111

This comment has been minimized.

Copy link
Author

commented Mar 7, 2015

@estesp They are two daemon in different vm.
I(or my boss?) want the container works in VLAN with other linux hosts,not a selected private address range.
I just want no conflict in ip address in this situation when container is created.either a warning or no ip adress assigned if conflict exists.Not a system down and handy find ip confilct.
It's the worst memory since I use docker.
In my view,docker's network manager(port mapping) is not so good for production.
some company even use -host for production.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2015

@shi154462671 For this reason there is also a --fixed-cidr flag for the daemon that you can use to make sure Docker doesn't hand out ip addresses you know are used by other things on the subnet.

@Johnsmith1111

This comment has been minimized.

Copy link
Author

commented Mar 7, 2015

@estesp So what about multi daemon on multi host?Am I should configure every docker daemon for different --fixed-cidr?

@estesp

This comment has been minimized.

Copy link
Contributor

commented Mar 10, 2015

@shi154462671 the best case for more complex multi-host/multi-daemon scenarios is to rely on SDN-like technology. Given socketplane.io is now part of the Docker team, I would expect you will see more about this very soon (see Madhu's note of the last 24 hours: https://groups.google.com/d/msg/docker-dev/vmJnCKfcA8Q/XB7nd2i80xwJ). There are also existing projects/products already covering this area, and I believe the expectation from a Docker engine perspective is this will not be solved by changes in the engine/daemon, but networking plugins will allow for much more interesting/complex scenarios where network coordination across many hosts and daemons is required.

@thaluska

This comment has been minimized.

Copy link

commented Jul 22, 2015

Hello, I have the same problem from docker 1.6 up. In docker 1.5 this configuration work. I have VM running ubuntu 14.04 which is in network subnet 192.168.1.0/24. 192.168.1.1 is set for KVM gateway. VM which is running ubuntu and is a docker host has IP 192.168.1.2/24 this IP I assign to manual created bridge0.
It is setup in /etc/network/interfaces
auto eth0
iface eth0 inet manual
auto bridge0
iface bridge0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1
bridge_ports eth0

In /etc/default/docker I set DOCKER_OPTS="-b bridge0" i also try DOCKER_OPTS="-b=bridge0" or DOCKER_OPTS="--bridge=bridge0".

After this I restart the docker daemon. A run first docker container like docker run -it --name test ubuntu /bin/bash. Problem is that this container get IP 192.168.1.1 which is already assigned. When I remove the container and create another then the docker correctly skip the bridge IP. But the gateway IP conflict when the docker daemon has been restarted and the docker IP count was reset.

This start happening after I upgrade docker from 1.6 or if I use also docker 1.7.1.
Thank you

@thomasshanks

This comment has been minimized.

Copy link

commented Aug 20, 2015

I would like a way to give docker an entire /24 subnet except for the IP of the default gateway, which is in this case a router that is plugged into an interface bridged onto docker0. We have large customers who would also be interested in this.

A good way to avoid containers stomping on someone already using the IP would be to just arping on the network and see if the IP is already in use. That seems like a natural and perfectly reasonable thing to do. Why not do that?

There is, of course, the chance that the user of the in-use IP could be temporarily down at the time this check occurs. It would make sense to also provide a way to statically exclude part of the IP range to avoid a problem in this situation.

One way to do this would be for docker to look up all of the IPs currently listed in routes, or at least used in the default route, and not allocate those IPs to containers. You already skip the bridge IP. Why not skip the default gateway IP as well?

Another way you could do this is to change the behavior of --fixed-cidr to skip allocating IPs less than the specified IP. If you specified 10.10.10.2/24, it would skip the offending gateway IP, 10.10.10.1.

@estesp

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2015

Now that (as of Docker 1.7) the networking implementation is handled in the project docker/libnetwork I would suggest this issue or whatever part of it is of interest to users be added as an issue there so that the libnetwork team can interact on what the issue or requested feature is. Also, bridge mode is quickly becoming one of many options for network management, as in Docker 1.9, the currently experimental Docker advanced networking features are most likely going to move to the stable release binary, so some use cases may find an overlay network scenario more useful for their network management paradigm.

ping @mrjana @mavenugo

@aboch

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2015

In libnetwork we are working to provide an IPAM driver which the user would somehow be able to program before bridge driver starts allocating IP for the containers.

It's work in progress and current proposal is not complete yet.
You can check progress at libnetwork/pull/459.

@thomasshanks

This comment has been minimized.

Copy link

commented Aug 21, 2015

Until then, could you at least make the obvious change of checking if an IP is in use before you assign it to a container, or of excluding the gateway IP from allocation?

@mrjana

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2015

@thomasshanks Have you looked --default-gateway option? It seems like it be a good fit for your use case. This option provides you a way to assign a default gateway which is different from your bridge IP and docker will not try to allocate that IP address for a container. And you can use --fixed-cidr to tell docker to only allocate from that range. So I believe we already have all the knobs for you to do this yourself.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Oct 22, 2015

Is this one ready to close, now that the IPAM driver PR is merged (docker/libnetwork#525)?

@sitamet

This comment has been minimized.

Copy link

commented Jan 9, 2016

I'm experiencing IP conflicts: two containers with same IP!

I'm running docker with default options:

docker info
Containers: 391
Images: 586
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1368
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 62.89 GiB

1st docker inspect network settings:

    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "325d461067932cb180061cea19a9fd55f936433a8ed9bcdbff80880213fdd818",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "3306/tcp": null
        },
        "SandboxKey": "/var/run/docker/netns/325d46106793",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "4db1346b0504e49906fb0f9b5ecae4080fee2fbbdd8ac3560c77358e70e4dd72",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.126",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:7e",
        "Networks": {
            "bridge": {
                "EndpointID": "4db1346b0504e49906fb0f9b5ecae4080fee2fbbdd8ac3560c77358e70e4dd72",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.126",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:7e"
            }
        }
    }

second container:

    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "5acd6b64b7b97939f17bcf407be2bfb112e18cb0791ef2363044760c23fc0025",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "443/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "443"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "80"
                }
            ]
        },
        "SandboxKey": "/var/run/docker/netns/5acd6b64b7b9",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "2cace3c896fb303ef135e1e8cb6a99615e4f233e52896f5952c84fcf5d8fd9c8",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.126",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:7e",
        "Networks": {
            "bridge": {
                "EndpointID": "2cace3c896fb303ef135e1e8cb6a99615e4f233e52896f5952c84fcf5d8fd9c8",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.126",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:7e"
            }
        }
    }
@mavenugo

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2016

@sitamet it is a duplicate of #18535 and is already resolved in master.

@thaJeztah yes. we can close this issue.

@thaJeztah thaJeztah closed this Jan 9, 2016

@pejas

This comment has been minimized.

Copy link

commented Mar 2, 2016

Can we reopen this issue it still happens with Docker version 1.10.2, build c3959b1?
I have a bridge:
cat /etc/sysconfig/network-scripts/ifcfg-br0
TYPE="Bridge"
BOOTPROTO="static"
DEVICE="br0"
ONBOOT="yes"
IPADDR="192.168.10.5"
PREFIX="24"
GATEWAY="192.168.10.1"
DNS1="192.168.10.1"

cat /etc/sysconfig/network-scripts/ifcfg-enp11s0f0
TYPE="Ethernet"
NAME="enp11s0f0"
UUID="EDITED"
DEVICE="enp11s0f0"
ONBOOT=yes
BRIDGE=br0

which is used by host (default route) and kvm virtual machines
ip route show
default via 192.168.10.1 dev br0

When vm starts linux/windows/anything gets an ip assigned by dhcp, which I can change afterwords to static if I want to.

When I start docker with param --bridge="br0" first container gets 192.168.10.1 - my gateway!

Why and how? It wasn't assigned via dhcp, I didn't specify it anywhere, docker just made it up and set as static with ip?

Issue #18535 was merged before 1.10.0 so that fix didn't help.

@mavenugo

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2016

@pejas i dont think it is related. br0 ip-address is not 192.168.10.1 and it is of the gateways and the docker ipam driver will not work with such external configurations.
if you prefer the IPAM to not allocate such addresses, then you can use the --fixed-cidr option to allocate addresses from a different range. If there is no --fixed-cidr, it will use the entire subnet range to allocate the addresses.

You can read more about it : https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/

@AhmadFazliIsmail

This comment has been minimized.

Copy link

commented Sep 27, 2017

Sorry to re-open this thread. I am also having experience to this issue since last week and now just got the solution. Just to share my successful setup and say thank you to https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/ and https://docs.docker.com/engine/userguide/networking/#use-a-proxy-server-with-containers.

My physical host has 1 NIC, then I create a bridge named br0 (IPv4 192.168.0.101) which host 3 VMs and 3 Docker containers. All VMs and Docker containers have IP address in subnet 192.168.0.0 and able to communicate each other (VM can ping Docker container and vice versa). May be this is what @Johnsmith1111 required in order to prevent IP address conflict.

Below are the configurations those were used;

In /etc/docker/daemon.json;

{
  "iptables": true,
  "bip": "172.17.0.1/16",
  "ipv6": false,
  "fixed-cidr": "172.17.0.1/16",
  "dns": ["8.8.8.8","8.8.4.4"],
  "data-root": "/var/lib/docker/",
  "storage-driver": "devicemapper",
  "selinux-enabled": false
}

Then I create another Docker network bridge using this command;
docker network create --driver=bridge --subnet="192.168.0.0/24" --ip-range="192.168.0.10/24" --gateway="192.168.0.101" --aux-address "DefaultGatewayIPv4"="192.168.0.1" -o "com.docker.network.bridge.default_bridge"="false" -o "com.docker.network.bridge.enable_icc"="true" -o "com.docker.network.bridge.enable_ip_masquerade"="false" -o "com.docker.network.bridge.host_binding_ipv4"="0.0.0.0" -o "com.docker.network.bridge.name"="br0" -o "com.docker.network.driver.mtu"="1500" dockerbr0

To run a Docker container, use this command;
docker run --network dockerbr0 --ip 192.168.0.10 --privileged -d -h test --name test --rm --memory="256m" --memory-swap="256m" --oom-kill-disable --cpus="0.5" my-own-registry:5000/centos7:test

The most important thing is to define --ip 192.168.0.xxx which should not already existed in your local network! Hope this configuration able to help someone with this issue.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Sep 27, 2017

@AhmadFazliIsmail you can also exclude IP-addresses from a network using the --aux-address option on docker network create. For example, the following shows that IP-addresses 192.168.1.2 and 192.168.1.3 should be excluded;

docker network create \
  --ip-range=192.168.1.0/24 \
  --subnet=192.168.1.0/16 \
  --aux-address="my-router=192.168.1.2" \
  --aux-address="my-switch=192.168.1.3" \
  foo

Now, when starting three containers, those IP-addresses are skipped and not assigned to a container;

docker run -dit --network=foo --name=c1 busybox
docker run -dit --network=foo --name=c2 busybox
docker run -dit --network=foo --name=c3 busybox


docker network inspect foo --format '{{ json .Containers }}'
{
  "f4df6d4f8e3d39fdf316a32e06baadaa9d2c3d8e7029c585416aea648f04897b": {
    "Name": "c1",
    "EndpointID": "800b1191a6ea7690a72f1c41d1c437b8cf0a219a097a7827fff97f74265330d2",
    "MacAddress": "02:42:c0:a8:01:01",
    "IPv4Address": "192.168.1.1/16",
    "IPv6Address": ""
  },
  "2abec18a8296028b370abde24450870834936df16028b55eb6cc8e76d84e3b7b": {
    "Name": "c2",
    "EndpointID": "7d51f69001f14a3bd59154e9121df5acbe33bdee5c0b760ba9b55e49dc1c0553",
    "MacAddress": "02:42:c0:a8:01:04",
    "IPv4Address": "192.168.1.4/16",
    "IPv6Address": ""
  },
  "73babd22fb712d85421d61c8ea2447d243e1a86a5ab4e140e6a6d3cb2a6859a5": {
    "Name": "c3",
    "EndpointID": "b53813007ad73d609470b969791269742707c2fc708ab99307311e6a9f0946b2",
    "MacAddress": "02:42:c0:a8:01:05",
    "IPv4Address": "192.168.1.5/16",
    "IPv6Address": ""
  }
}
@e-ruiz

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.