Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add daemon options for Docker network subnets #21776

Closed
dimaspivak opened this issue Apr 5, 2016 · 57 comments
Closed

Add daemon options for Docker network subnets #21776

dimaspivak opened this issue Apr 5, 2016 · 57 comments

Comments

@dimaspivak
Copy link

@dimaspivak dimaspivak commented Apr 5, 2016

Output of docker version:

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 14
 Running: 9
 Paused: 0
 Stopped: 5
Images: 7
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Backing Filesystem: xfs
 Dirs: 63
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 120.1 GiB
Name: my-machine-hostname
ID: PN5G:4THA:2ZTC:OF6B:BNMQ:MSB7:JCUK:TBAW:3ZRP:ARMT:ABWV:YFYU
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Happens on physical hosts and AWS/GCP

Steps to reproduce the issue:

  1. Set --bip=... when starting a Docker daemon (e.g. to avoid conflicts with other things on your company's LAN). I set mine to 192.168.1.0/16.
  2. Create a Docker bridge network (e.g. docker network create ...).
  3. Run docker network inspect against the new network and look at the subnet.

Describe the results you received:
The subnet on my machine is, for example, 172.19.0.0/16

Describe the results you expected:
Seeing as how I specified a default bridge IP for the daemon, I'd have expected that to be propagated to the bridge driver.

Additional information you deem important (e.g. issue happens only occasionally):
An obvious workaround is to use --subnet when creating the bridge networks. The problem, though, is that this can lead to race conditions if networks are started in multiple threads. That is, I end up writing a lot of exception handling for cases of overlapping subnet ranges, but it seems more straightforward for Docker to handle this (as it already does on the default subnet it seems to hardcode).

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 5, 2016

--bip only affects the docker0 bridge, which is for the default network only.

@dimaspivak
Copy link
Author

@dimaspivak dimaspivak commented Apr 5, 2016

Yeah, that's what my experimentation led to me to understanding. Perhaps this is more of a feature request, in that case, to allow specification of a default subnet from which to start looking for open ones when creating new bridge networks?

@holgerreif
Copy link

@holgerreif holgerreif commented Apr 6, 2016

This is a feature request and I support it!

docker daemon should have two additional options:

  • --net-cdir specifies an ip range from which docker network create should get their addresses from. This is similiar to the --bip option for traditional docker0 bridge w/o specifying the IP for the dockerhost on the bridge.
  • --net-size spedifies the size of a docker network created network via its subnet len.
    Thus we can have automatic adress assigment w/o clashes with existing networks.

Currently we have to manage ip subnets for networks manually, which is especially ugly when using docker-compose. The reason is basically the same as mentioned in #21329: not keeping track of networks.

Example:

--net-cdir=10.1.0.0/8 --net-size=24 would select 10.1.0.0/24 for the first network, 10.1.1.0/24 for the second one and so on. Conflicts with existing networks should be checked as usual, overlapping with --bip Parameter should be allowed.

Instead of --net-cdir an alternative name for this option would be --net-pool.

@wenchma
Copy link
Contributor

@wenchma wenchma commented Apr 7, 2016

@holgerreif --net-cidr and --net-size two additaional options will be welcomed to specify the subnet pool, I also need this feature.

@mavenugo
Copy link
Contributor

@mavenugo mavenugo commented Apr 7, 2016

@dimaspivak @wenchma @holgerreif As you might know already docker network create already supports --subnet & --ip-range options just for this purpose. What is the purpose of adding more flags to the daemon just for the sake of defining pre-defined pools (for automatic assignment ? ).

@dimaspivak
Copy link
Author

@dimaspivak dimaspivak commented Apr 7, 2016

Exactly, I see this as a daemon option (or, if such a thing exists, as a host-level option for the bridge network driver). In my case, I create a half-dozen bridge networks simultaneously. If I leave out --subnet when starting each, I have the desired behavior where they pick their own subnet, but this seems hard-coded to start looking in the 172... range that conflicts with other hosts on my local network. If I include --subnet, I end up having to build a not-insignificant amount of exception handling to recognize when the client complains about the requested subnet overlapping with an existing one, at which time it then has to try again (potentially hitting other overlaps along the way) until it gets an available one. Essentially, I'm saying that Docker already supports the correct behavior, but simply seems to have hardcoded the IP range to use when starting its search for free pools.

@holgerreif
Copy link

@holgerreif holgerreif commented Apr 7, 2016

Actually it's a PITA to manage/select subnets manually.
OTOH I usually do not care about the actual ip addrs choosen, as long as they stick to a defined network range not conflicting with the rest of the network far outside docker host.

BTW: I know the (note from userguide)[https://docs.docker.com/engine/userguide/networking/work-with-networks/#create-networks] It is highly recommended to use the --subnet option while creating a network. But defining a single range to take networks from once is much more pleasure than to do the selection every time you actually create a network (or let docker-compose create one for you).

As @dimaspivak already stated: the logic to pick the right subnet is already there! You just need to feed the right params (starting point --net-pool and size --net-size)

As a goodie (I guess this is just feeding the network select logic with specific parameter to override the global option) one can image to add the parameter --net-size to docker network create as well. And as an option of docker-compose file too.
Well, this is another, more complex issue, since this goodie would require to change the docker remote API...

@LukasBacigal
Copy link

@LukasBacigal LukasBacigal commented May 2, 2016

I guess we are trying to modify those pools.
docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go
it is not uncommon to use 172.17-31.x.x ... in company environment, and I faced problem of overlapping earlier.
so +1 for daemon feature to modify default pool.

@jameseck
Copy link

@jameseck jameseck commented May 24, 2016

I would also like to +1 for this feature. I'm in the same situation where new networks created with "docker network create" or docker-compose, etc still use 172.16.x.x and this address range is already in use by other internal services that I need to access.

@nschoe
Copy link

@nschoe nschoe commented May 25, 2016

I'll +1 this too, as explained in the issue mentioned above, I think we need:

  1. to be able to specify the subnet
  2. to be able to specify the prefix-length (in my case, I don't really care if this is in 172.x, but I do care that it creates /16 subnets by default as I'm greatly limited, and I'd prefer to use /24)

So +1 ^^

@mavenugo
Copy link
Contributor

@mavenugo mavenugo commented May 27, 2016

@aboch can you PTAL

@jpetazzo
Copy link
Contributor

@jpetazzo jpetazzo commented May 28, 2016

Suggestion: as a workaround, if you want to prevent Docker from using specific subnets, you can do:

docker network create --subnet 172.16.0.0/24 reserved-172-16-0-0-24

Then that subnet won't be used by Docker when auto-allocating address space to other networks.

@codablock
Copy link

@codablock codablock commented Jun 1, 2016

We have the same problem with overlapping networks, causing random network failures on our CI servers. We also can not use the --subnet option from "docker network create" due to the custom clash avoidance which would be needed.

@jpetazzo Did you test your workaround? Wouldn't it still cause routing failures due to duplicate/clashing routes?

@shane-axiom
Copy link

@shane-axiom shane-axiom commented Jun 3, 2016

+1, badly needed! docker network create is creating networks which override active subnets in our physical network.

@shane-axiom
Copy link

@shane-axiom shane-axiom commented Jun 3, 2016

@jpetazzo @codablock I tried the suggested workaround, but as @codablock mentioned this still causes routing failures with existing non-docker subnets. If you're just trying to blacklist an arbitrary subnet from being used by docker network create this will work (because it'll already be in use by another docker network), but if your problem is docker creating subnets that conflict with existing subnets on your network this isn't a viable workaround.

@codablock
Copy link

@codablock codablock commented Jun 6, 2016

Can anyone of the Docker team give an estimate if a fix (or a new feature) for this could be implemented for 1.12 or 1.13? Depending on this we could decide how much effort to put into a workaround.

@justincormack
Copy link
Contributor

@justincormack justincormack commented Jun 6, 2016

@codablock The freeze for features for 1.12 is June 14, so it is going to be tight to get a fix in by then, and I am not aware of anyone working on it. Of course if anyone wants to we will make best efforts to get it in for 1.12, as it is obviously a serious issue for some users.

@Kodiack
Copy link

@Kodiack Kodiack commented Jul 5, 2016

@martynsmith found a nice workaround today after reading some of Docker's source code. Docker will not create networks that conflict with nameserver entries in /etc/resolv.conf, so adding a bogus entry there can prevent Docker from creating networks in a subnet that may already be used for something else.

In our case, the 172.17.0.0/16 subnet would conflict with a VPN, so people could not connect to their systems or other internal services if Docker was installed on a system with the default network in use. We were able to change the docker0 range with the --bip argument in /etc/default/docker, but creating new Docker networks (e.g. with Compose) could cause conflicts to arise again.

The fairly simple workaround in our case involved adding this line to /etc/resolv.conf:
nameserver 172.17.254.254

With that line, Docker no longer creates new networks in the 172.17.0.0/16 subnet, saving us a good amount of headache when we haven't specifically defined how to set up a Docker network. Though it's also worth noting that Docker states that it's best to manually define a network (e.g. in a compose file), which is what I've been doing to this point (and will continue to do):

https://docs.docker.com/engine/userguide/networking/work-with-networks/#/create-networks

"Note : It is highly recommended to use the --subnet option while creating a network. If the --subnet is not specified, the docker daemon automatically chooses and assigns a subnet for the network and it could overlap with another subnet in your infrastructure that is not managed by docker. Such overlaps can cause connectivity issues or failures when containers are connected to that network."

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

@LukasBacigal
Copy link

@LukasBacigal LukasBacigal commented Jul 14, 2016

Should we open another issue as feature request? or change labels to feature and unlabel v 1.10

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Jul 14, 2016

(version) Labels are automatically set based on the version supplied in the original issue, no need to change.

@creynders
Copy link

@creynders creynders commented Sep 21, 2016

👍 💯

@panga
Copy link

@panga panga commented Oct 3, 2016

I'm having the same issue with a corporate network, please add an option to change default subnets (+1)

@robertknox
Copy link

@robertknox robertknox commented Jul 7, 2017

I tried the /etc/resolv.conf tech. It did not work.
Is there anyway to tell Docker not to hand out a certain address within the subnet mask?
In my situation I have a bridge to an external net. I need to talk to one external machine at 172.27.153.1.
( 172.27.153.0/24 ) The very first address Docker hands out when starting a new container is 172.27.153.1. I also gives this address to the the host bridge as well. So I can no longer talk to the external 172.27.153.1.
I am stuck. Any suggestions?

@jmkgreen
Copy link

@jmkgreen jmkgreen commented Jul 7, 2017

Note: We're using Docker 17.06.ce with Swarm.

Please see https://docs.docker.com/engine/userguide/networking/get-started-overlay/#create-the-overlay-network which includes a really important recommendation to always specify the desired subnet of your new network.

We think Docker's position is that if you have non-Docker private subnets, you absolutely must create subnets for Docker's own use manually and configure them all to not conflict.

Also be advised that in our experience, when using the overlay network driver, Docker will consume IPs for it's own use in addition to the ones your declared containers will use. This has caused us to calculate the subnet size based on the expected number of containers then double it, else stacks will not start fully and they provide no feedback as to why.

@robertknox
Copy link

@robertknox robertknox commented Jul 7, 2017

Thanks very much for the response.

I am creating a subnet for Docker. The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.) If I pass docker a subnet of 172.27.153.0/24, the very first container created will get 172.27.153.1.

I am not using the overlay driver with swarm. I am bridging to a linux bridge on the host. This allows me access to a subnet (external) and the all important 172.27.153.1 box.

I have tried putting in 172.27.153.1 into /etc/resolv.conf on the host machine as described above to force docker not to use that ip when handing out ips -- no luck. Is there another way to do this?

I am trying to think of options: I have read that I can pass into docker "run" an ip address:"docker run --ip XXXXXX". Maybe that is my option? I could manage all the ips myself, handing out an ip I know is safe at "docker run time"? I need to create 200 docker container's each acting as a certain type of device that will interact with the 172.27.153.1 box that sits on the network.

I very much appreciate your help on this issue.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Jul 7, 2017

@robertknox You can specify the range of IP's to use beyond just the subnet. On network create it's --ip-range, for the default bridge you set it using --fixed-cidr in the daemon options.

@robertknox
Copy link

@robertknox robertknox commented Jul 7, 2017

thanks very much for your suggestion, but I am not using the default bridge. I am using a custom bridge to a linux bridge to a particular port on my machine. I am passing in the --ip-range option, but it gives me just that -- a range. I cannot say "don't use this one ip address."

docker network create --driver=bridge --ip-range=172.27.153.0/24 --subnet=172.27.153.0/24 -o "com.docker.network.bridge.name=br0" br0

The first container created will get the ip = 172.27.153.1 and that conflicts with another box on this network.

Thanks again.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Jul 7, 2017

@robertknox
Copy link

@robertknox robertknox commented Jul 7, 2017

I didn't know --ip-range can be used multiple times to indicate the ip nums slice desired.
thanks so much

@robertknox
Copy link

@robertknox robertknox commented Jul 7, 2017

Could you give an example?

@aboch
Copy link
Contributor

@aboch aboch commented Jul 8, 2017

@robertknox

The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.)

--aux-address is the option to tell IPAM which addresses are already taken in your network. So they won't be given out to containers.

Example:

docker network create --subnet 12.12.0.0/24 --aux-address dns=12.12.0.53 --aux-address dut=12.12.0.72 --gateway 12.12.0.254 my_phy_network
@robertknox
Copy link

@robertknox robertknox commented Jul 8, 2017

Thanks very much

@archenroot
Copy link

@archenroot archenroot commented Aug 9, 2017

I got it working on Centos 7.3 with Docker 17.06.0 CE with just small issues:
#34461

@andypost
Copy link

@andypost andypost commented Sep 16, 2017

btw Why not make subnets smaller? for example using mask 8 we can get *8 subnets 172.19.0.0/8

@zarbis
Copy link

@zarbis zarbis commented Sep 16, 2017

@andypost How exactly will it solve problem of subnet shortage?

@rdxmb
Copy link

@rdxmb rdxmb commented Oct 4, 2017

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

This seems to be the best solution for me.
And instead of defining subnets via daemon-cli, this could (and should) be done via config-File in /etc/docker like https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/

@batmat
Copy link

@batmat batmat commented Jan 3, 2018

The workaround with resolv.conf above did not work for me using Docker 17.12.0-ce. The only thing that did was using @jpetazzo's workaround + deleting the associated route, which is quite hackish. Also because this fake network will get wiped out each time one runs docker network prune...

TL;DR (if I want to prevent 172.x.x.x usage, which includes especially some of our VPN range):

docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0

Illustrated:

$ docker network create network$RANDOM; docker network create network$RANDOM
605ad9cd3ef6f1572d421b6de3d730af594cbd26b90c4471100921e9c00221dd
9f453f5b28d2a384b73f5d259e2dc366da9d59f94a60776e6ba091287a648a97
$ docker network ls                                                         
NETWORK ID          NAME                DRIVER              SCOPE
3b34008790f1        bridge              bridge              local
67b97edbee5e        host                host                local
9f453f5b28d2        network16470        bridge              local
605ad9cd3ef6        network7807         bridge              local
bd0efb23366a        none                null                local
$ ifconfig | grep br- -A1                                                   
br-605ad9cd3ef6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
--
br-9f453f5b28d2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
$ docker network prune     # above you see the 172.18.0.1 clash I do not want
WARNING! This will remove all networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
network7807
network16470

$ docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
63c91b810977cd883a8f03e93f43c5c55c5dd3b040aeda18623895ce657fe0ae
$ route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0
SIOCDELRT: Operation not permitted
$ sudo route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0                     7
$ docker network create network$RANDOM; docker network create network$RANDOM
7754974a0a2c853a83913a760f797e504fbc9b8ca3fe015d1b7862d9f602a2ad
2b93f62df1470d8ae36a981a24b3ebea9308172364fa0e465c0ac948fd71856d
$ ifconfig | grep br- -A1
br-2b93f62df147: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.32.1  netmask 255.255.240.0  broadcast 192.168.47.255
--
br-63c91b810977: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.0.0.1  netmask 255.0.0.0  broadcast 172.255.255.255
--
br-7754974a0a2c: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.16.1  netmask 255.255.240.0  broadcast 192.168.31.255

Also: FTR, the solution involving using explicit parameters does not work for my case as I'm using docker-compose. So I could probably modify every docker-compose.yml I'm using out there, but I suspect that would slightly unideal :P.

I'd welcome any hint for a less-hackish/cleaner solution or workaround. Thank you everyone and thanks again for that great tool that is Docker!

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 16, 2018

@moisei Pull-request #29376 was closed, not merged, so that feature is not available (yet) in Docker

@orodbhen
Copy link

@orodbhen orodbhen commented Jan 18, 2018

This would be hugely useful. I'm dealing with messaging software that requires all member host IPs to be recorded in a configuration file. The simplest way I can find of doing this is creating the network, querying Docker for the subnet, and dumping all 4096 possible IPs into the file. That's likely going to create performance issues for the messaging software, and a lot of extra traffic.

I can't know what the container IPs will be until they've started up, so this appears to be the only way.

If I set the subnet explicitly, then I have to code up all the logic to prevent overlap.

It would be so much better to be able to set the subnet size in the compose file.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 19, 2018

It would be so much better to be able to set the subnet size in the compose file.

You can already set network options (such as subnet) in the compose-file; https://docs.docker.com/compose/compose-file/#network-configuration-reference

The feature requested here is to set a default on the daemon, that's used if you did not specify those options when creating a network

@orodbhen
Copy link

@orodbhen orodbhen commented Jan 19, 2018

@thaJeztah There's also some discussion regarding making the option available from the
client side as well. Either or both would be very helpful.

You can set the subnet explicitly in the compose file, but there's no
option to simply set the size and let Docker select the address.

Regardless of how it's done, the idea is to avoid having to validate the
user-provided subnet to prevent overlap.

Incidentally, though, there is an easy way to do this in Python with the
ipaddress module.

net1 = ip_address(subnet1)
net2 = ip_address('subnet2)
invalid = net1.overlaps(net2)
@cristianocasella
Copy link

@cristianocasella cristianocasella commented Feb 8, 2018

hi guys, I put a PR for this trouble: moby/libnetwork#2072

@marcoreni
Copy link

@marcoreni marcoreni commented Mar 22, 2018

Hello @cristianocasella , looks like a PR ( #36396 ) is almost ready for merge and should allow the user to specify one or more address pool, thus guaranteeing more freedom.

@Yajo
Copy link

@Yajo Yajo commented May 29, 2018

#36396 is merged, shouldn't this be closed?

@cristianocasella
Copy link

@cristianocasella cristianocasella commented May 29, 2018

Thanks for your support. With #36396 this can be closed.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented May 29, 2018

Thanks!

@cpuguy83 cpuguy83 closed this May 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.