Add daemon options for Docker network subnets

[dimaspivak]

Output of docker version:

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 14
 Running: 9
 Paused: 0
 Stopped: 5
Images: 7
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Backing Filesystem: xfs
 Dirs: 63
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 120.1 GiB
Name: my-machine-hostname
ID: PN5G:4THA:2ZTC:OF6B:BNMQ:MSB7:JCUK:TBAW:3ZRP:ARMT:ABWV:YFYU
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Happens on physical hosts and AWS/GCP

Steps to reproduce the issue:

1. Set --bip=... when starting a Docker daemon (e.g. to avoid conflicts with other things on your company's LAN). I set mine to 192.168.1.0/16.
2. Create a Docker bridge network (e.g. docker network create ...).
3. Run docker network inspect against the new network and look at the subnet.

Describe the results you received:
The subnet on my machine is, for example, 172.19.0.0/16

Describe the results you expected:
Seeing as how I specified a default bridge IP for the daemon, I'd have expected that to be propagated to the bridge driver.

Additional information you deem important (e.g. issue happens only occasionally):
An obvious workaround is to use --subnet when creating the bridge networks. The problem, though, is that this can lead to race conditions if networks are started in multiple threads. That is, I end up writing a lot of exception handling for cases of overlapping subnet ranges, but it seems more straightforward for Docker to handle this (as it already does on the default subnet it seems to hardcode).

[cpuguy83]

--bip only affects the docker0 bridge, which is for the default network only.

[dimaspivak]

Yeah, that's what my experimentation led to me to understanding. Perhaps this is more of a feature request, in that case, to allow specification of a default subnet from which to start looking for open ones when creating new bridge networks?

[holgerreif]

This is a feature request and I support it!

docker daemon should have two additional options:

• --net-cdir specifies an ip range from which docker network create should get their addresses from. This is similiar to the --bip option for traditional docker0 bridge w/o specifying the IP for the dockerhost on the bridge.
• --net-size spedifies the size of a docker network created network via its subnet len.
  Thus we can have automatic adress assigment w/o clashes with existing networks.

Currently we have to manage ip subnets for networks manually, which is especially ugly when using docker-compose. The reason is basically the same as mentioned in #21329: not keeping track of networks.

Example:

--net-cdir=10.1.0.0/8 --net-size=24 would select 10.1.0.0/24 for the first network, 10.1.1.0/24 for the second one and so on. Conflicts with existing networks should be checked as usual, overlapping with --bip Parameter should be allowed.

Instead of --net-cdir an alternative name for this option would be --net-pool.

[wenchma]

@holgerreif --net-cidr and --net-size two additaional options will be welcomed to specify the subnet pool, I also need this feature.

[mavenugo]

@dimaspivak @wenchma @holgerreif As you might know already docker network create already supports --subnet & --ip-range options just for this purpose. What is the purpose of adding more flags to the daemon just for the sake of defining pre-defined pools (for automatic assignment ? ).

[dimaspivak]

Exactly, I see this as a daemon option (or, if such a thing exists, as a host-level option for the bridge network driver). In my case, I create a half-dozen bridge networks simultaneously. If I leave out --subnet when starting each, I have the desired behavior where they pick their own subnet, but this seems hard-coded to start looking in the 172... range that conflicts with other hosts on my local network. If I include --subnet, I end up having to build a not-insignificant amount of exception handling to recognize when the client complains about the requested subnet overlapping with an existing one, at which time it then has to try again (potentially hitting other overlaps along the way) until it gets an available one. Essentially, I'm saying that Docker already supports the correct behavior, but simply seems to have hardcoded the IP range to use when starting its search for free pools.

[holgerreif]

Actually it's a PITA to manage/select subnets manually.
OTOH I usually do not care about the actual ip addrs choosen, as long as they stick to a defined network range not conflicting with the rest of the network far outside docker host.

BTW: I know the (note from userguide)[https://docs.docker.com/engine/userguide/networking/work-with-networks/#create-networks] It is highly recommended to use the --subnet option while creating a network. But defining a single range to take networks from once is much more pleasure than to do the selection every time you actually create a network (or let docker-compose create one for you).

As @dimaspivak already stated: the logic to pick the right subnet is already there! You just need to feed the right params (starting point --net-pool and size --net-size)

As a goodie (I guess this is just feeding the network select logic with specific parameter to override the global option) one can image to add the parameter --net-size to docker network create as well. And as an option of docker-compose file too.
Well, this is another, more complex issue, since this goodie would require to change the docker remote API...

[LukasBacigal]

I guess we are trying to modify those pools.
docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go
it is not uncommon to use 172.17-31.x.x ... in company environment, and I faced problem of overlapping earlier.
so +1 for daemon feature to modify default pool.

[jameseck]

I would also like to +1 for this feature. I'm in the same situation where new networks created with "docker network create" or docker-compose, etc still use 172.16.x.x and this address range is already in use by other internal services that I need to access.

[nschoe]

I'll +1 this too, as explained in the issue mentioned above, I think we need:

1. to be able to specify the subnet
2. to be able to specify the prefix-length (in my case, I don't really care if this is in 172.x, but I do care that it creates /16 subnets by default as I'm greatly limited, and I'd prefer to use /24)

So +1 ^^

[mavenugo]

@aboch can you PTAL

[jpetazzo]

Suggestion: as a workaround, if you want to prevent Docker from using specific subnets, you can do:

docker network create --subnet 172.16.0.0/24 reserved-172-16-0-0-24

Then that subnet won't be used by Docker when auto-allocating address space to other networks.

[codablock]

We have the same problem with overlapping networks, causing random network failures on our CI servers. We also can not use the --subnet option from "docker network create" due to the custom clash avoidance which would be needed.

@jpetazzo Did you test your workaround? Wouldn't it still cause routing failures due to duplicate/clashing routes?

[srstsavage]

+1, badly needed! docker network create is creating networks which override active subnets in our physical network.

[srstsavage]

@jpetazzo @codablock I tried the suggested workaround, but as @codablock mentioned this still causes routing failures with existing non-docker subnets. If you're just trying to blacklist an arbitrary subnet from being used by docker network create this will work (because it'll already be in use by another docker network), but if your problem is docker creating subnets that conflict with existing subnets on your network this isn't a viable workaround.

[codablock]

Can anyone of the Docker team give an estimate if a fix (or a new feature) for this could be implemented for 1.12 or 1.13? Depending on this we could decide how much effort to put into a workaround.

[justincormack]

@codablock The freeze for features for 1.12 is June 14, so it is going to be tight to get a fix in by then, and I am not aware of anyone working on it. Of course if anyone wants to we will make best efforts to get it in for 1.12, as it is obviously a serious issue for some users.

[Kodiack]

@martynsmith found a nice workaround today after reading some of Docker's source code. Docker will not create networks that conflict with nameserver entries in /etc/resolv.conf, so adding a bogus entry there can prevent Docker from creating networks in a subnet that may already be used for something else.

In our case, the 172.17.0.0/16 subnet would conflict with a VPN, so people could not connect to their systems or other internal services if Docker was installed on a system with the default network in use. We were able to change the docker0 range with the --bip argument in /etc/default/docker, but creating new Docker networks (e.g. with Compose) could cause conflicts to arise again.

The fairly simple workaround in our case involved adding this line to /etc/resolv.conf:
nameserver 172.17.254.254

With that line, Docker no longer creates new networks in the 172.17.0.0/16 subnet, saving us a good amount of headache when we haven't specifically defined how to set up a Docker network. Though it's also worth noting that Docker states that it's best to manually define a network (e.g. in a compose file), which is what I've been doing to this point (and will continue to do):

https://docs.docker.com/engine/userguide/networking/work-with-networks/#/create-networks

"Note : It is highly recommended to use the --subnet option while creating a network. If the --subnet is not specified, the docker daemon automatically chooses and assigns a subnet for the network and it could overlap with another subnet in your infrastructure that is not managed by docker. Such overlaps can cause connectivity issues or failures when containers are connected to that network."

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

[LukasBacigal]

Should we open another issue as feature request? or change labels to feature and unlabel v 1.10

[cpuguy83]

(version) Labels are automatically set based on the version supplied in the original issue, no need to change.

[creynders]

👍 💯

[panga]

I'm having the same issue with a corporate network, please add an option to change default subnets (+1)

[robertknox]

I tried the /etc/resolv.conf tech. It did not work.
Is there anyway to tell Docker not to hand out a certain address within the subnet mask?
In my situation I have a bridge to an external net. I need to talk to one external machine at 172.27.153.1.
( 172.27.153.0/24 ) The very first address Docker hands out when starting a new container is 172.27.153.1. I also gives this address to the the host bridge as well. So I can no longer talk to the external 172.27.153.1.
I am stuck. Any suggestions?

[jmkgreen]

Note: We're using Docker 17.06.ce with Swarm.

Please see https://docs.docker.com/engine/userguide/networking/get-started-overlay/#create-the-overlay-network which includes a really important recommendation to always specify the desired subnet of your new network.

We think Docker's position is that if you have non-Docker private subnets, you absolutely must create subnets for Docker's own use manually and configure them all to not conflict.

Also be advised that in our experience, when using the overlay network driver, Docker will consume IPs for it's own use in addition to the ones your declared containers will use. This has caused us to calculate the subnet size based on the expected number of containers then double it, else stacks will not start fully and they provide no feedback as to why.

[robertknox]

Thanks very much for the response.

I am creating a subnet for Docker. The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.) If I pass docker a subnet of 172.27.153.0/24, the very first container created will get 172.27.153.1.

I am not using the overlay driver with swarm. I am bridging to a linux bridge on the host. This allows me access to a subnet (external) and the all important 172.27.153.1 box.

I have tried putting in 172.27.153.1 into /etc/resolv.conf on the host machine as described above to force docker not to use that ip when handing out ips -- no luck. Is there another way to do this?

I am trying to think of options: I have read that I can pass into docker "run" an ip address:"docker run --ip XXXXXX". Maybe that is my option? I could manage all the ips myself, handing out an ip I know is safe at "docker run time"? I need to create 200 docker container's each acting as a certain type of device that will interact with the 172.27.153.1 box that sits on the network.

I very much appreciate your help on this issue.

[cpuguy83]

@robertknox You can specify the range of IP's to use beyond just the subnet. On network create it's --ip-range, for the default bridge you set it using --fixed-cidr in the daemon options.

[robertknox]

thanks very much for your suggestion, but I am not using the default bridge. I am using a custom bridge to a linux bridge to a particular port on my machine. I am passing in the --ip-range option, but it gives me just that -- a range. I cannot say "don't use this one ip address."

docker network create --driver=bridge --ip-range=172.27.153.0/24 --subnet=172.27.153.0/24 -o "com.docker.network.bridge.name=br0" br0

The first container created will get the ip = 172.27.153.1 and that conflicts with another box on this network.

Thanks again.

[cpuguy83]

But `--ip-range` can be specified multiple times.

…

On Fri, Jul 7, 2017 at 1:06 PM, robertknox ***@***.***> wrote: thanks very much for your suggestion, but I am not using the default bridge. I am using a custom bridge to a linux bridge to a particular port on my machine. I am passing in the --ip-range option, but it gives me just that -- a range. I cannot say "don't use this one ip address." docker network create --driver=bridge --ip-range=172.27.153.0/24 --subnet= 172.27.153.0/24 -o "com.docker.network.bridge.name=br0" br0 The first container created will get the ip = 172.27.153.1 and that conflicts with another box on this network. Thanks again. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21776 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwxZifh4_9vJkIT-dZO6jQZOhAR0TwPks5sLmWRgaJpZM4IAgii> .

-- - Brian Goff

[robertknox]

I didn't know --ip-range can be used multiple times to indicate the ip nums slice desired.
thanks so much

[robertknox]

Could you give an example?

[aboch]

@robertknox

The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.)

--aux-address is the option to tell IPAM which addresses are already taken in your network. So they won't be given out to containers.

Example:

docker network create --subnet 12.12.0.0/24 --aux-address dns=12.12.0.53 --aux-address dut=12.12.0.72 --gateway 12.12.0.254 my_phy_network

[robertknox]

Thanks very much

[archenroot]

I got it working on Centos 7.3 with Docker 17.06.0 CE with just small issues:
#34461

[andypost]

btw Why not make subnets smaller? for example using mask 8 we can get *8 subnets 172.19.0.0/8

[zarbis]

@andypost How exactly will it solve problem of subnet shortage?

[rdxmb]

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

This seems to be the best solution for me.
And instead of defining subnets via daemon-cli, this could (and should) be done via config-File in /etc/docker like https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/

[batmat]

The workaround with resolv.conf above did not work for me using Docker 17.12.0-ce. The only thing that did was using @jpetazzo's workaround + deleting the associated route, which is quite hackish. Also because this fake network will get wiped out each time one runs docker network prune...

TL;DR (if I want to prevent 172.x.x.x usage, which includes especially some of our VPN range):

docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0

Illustrated:

$ docker network create network$RANDOM; docker network create network$RANDOM
605ad9cd3ef6f1572d421b6de3d730af594cbd26b90c4471100921e9c00221dd
9f453f5b28d2a384b73f5d259e2dc366da9d59f94a60776e6ba091287a648a97
$ docker network ls                                                         
NETWORK ID          NAME                DRIVER              SCOPE
3b34008790f1        bridge              bridge              local
67b97edbee5e        host                host                local
9f453f5b28d2        network16470        bridge              local
605ad9cd3ef6        network7807         bridge              local
bd0efb23366a        none                null                local
$ ifconfig | grep br- -A1                                                   
br-605ad9cd3ef6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
--
br-9f453f5b28d2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
$ docker network prune     # above you see the 172.18.0.1 clash I do not want
WARNING! This will remove all networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
network7807
network16470

$ docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
63c91b810977cd883a8f03e93f43c5c55c5dd3b040aeda18623895ce657fe0ae
$ route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0
SIOCDELRT: Operation not permitted
$ sudo route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0                     7
$ docker network create network$RANDOM; docker network create network$RANDOM
7754974a0a2c853a83913a760f797e504fbc9b8ca3fe015d1b7862d9f602a2ad
2b93f62df1470d8ae36a981a24b3ebea9308172364fa0e465c0ac948fd71856d
$ ifconfig | grep br- -A1
br-2b93f62df147: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.32.1  netmask 255.255.240.0  broadcast 192.168.47.255
--
br-63c91b810977: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.0.0.1  netmask 255.0.0.0  broadcast 172.255.255.255
--
br-7754974a0a2c: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.16.1  netmask 255.255.240.0  broadcast 192.168.31.255

Also: FTR, the solution involving using explicit parameters does not work for my case as I'm using docker-compose. So I could probably modify every docker-compose.yml I'm using out there, but I suspect that would slightly unideal :P.

I'd welcome any hint for a less-hackish/cleaner solution or workaround. Thank you everyone and thanks again for that great tool that is Docker!

[moisei]

what I am doing wrong?
https://stackoverflow.com/questions/48279424/default-address-pools-is-not-recognized-by-docker

[thaJeztah]

@moisei Pull-request #29376 was closed, not merged, so that feature is not available (yet) in Docker

[orodbhen]

This would be hugely useful. I'm dealing with messaging software that requires all member host IPs to be recorded in a configuration file. The simplest way I can find of doing this is creating the network, querying Docker for the subnet, and dumping all 4096 possible IPs into the file. That's likely going to create performance issues for the messaging software, and a lot of extra traffic.

I can't know what the container IPs will be until they've started up, so this appears to be the only way.

If I set the subnet explicitly, then I have to code up all the logic to prevent overlap.

It would be so much better to be able to set the subnet size in the compose file.

[thaJeztah]

It would be so much better to be able to set the subnet size in the compose file.

You can already set network options (such as subnet) in the compose-file; https://docs.docker.com/compose/compose-file/#network-configuration-reference

The feature requested here is to set a default on the daemon, that's used if you did not specify those options when creating a network

[orodbhen]

@thaJeztah There's also some discussion regarding making the option available from the
client side as well. Either or both would be very helpful.

You can set the subnet explicitly in the compose file, but there's no
option to simply set the size and let Docker select the address.

Regardless of how it's done, the idea is to avoid having to validate the
user-provided subnet to prevent overlap.

Incidentally, though, there is an easy way to do this in Python with the
ipaddress module.

net1 = ip_address(subnet1)
net2 = ip_address('subnet2)
invalid = net1.overlaps(net2)

[cristianocasella]

hi guys, I put a PR for this trouble: moby/libnetwork#2072

[marcoreni]

Hello @cristianocasella , looks like a PR ( #36396 ) is almost ready for merge and should allow the user to specify one or more address pool, thus guaranteeing more freedom.

[yajo]

#36396 is merged, shouldn't this be closed?

[cristianocasella]

Thanks for your support. With #36396 this can be closed.

[cpuguy83]

Thanks!