New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add daemon options for Docker network subnets #21776

Closed
dimaspivak opened this Issue Apr 5, 2016 · 57 comments

Comments

Projects
None yet
@dimaspivak

dimaspivak commented Apr 5, 2016

Output of docker version:

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 14
 Running: 9
 Paused: 0
 Stopped: 5
Images: 7
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Backing Filesystem: xfs
 Dirs: 63
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 120.1 GiB
Name: my-machine-hostname
ID: PN5G:4THA:2ZTC:OF6B:BNMQ:MSB7:JCUK:TBAW:3ZRP:ARMT:ABWV:YFYU
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Happens on physical hosts and AWS/GCP

Steps to reproduce the issue:

  1. Set --bip=... when starting a Docker daemon (e.g. to avoid conflicts with other things on your company's LAN). I set mine to 192.168.1.0/16.
  2. Create a Docker bridge network (e.g. docker network create ...).
  3. Run docker network inspect against the new network and look at the subnet.

Describe the results you received:
The subnet on my machine is, for example, 172.19.0.0/16

Describe the results you expected:
Seeing as how I specified a default bridge IP for the daemon, I'd have expected that to be propagated to the bridge driver.

Additional information you deem important (e.g. issue happens only occasionally):
An obvious workaround is to use --subnet when creating the bridge networks. The problem, though, is that this can lead to race conditions if networks are started in multiple threads. That is, I end up writing a lot of exception handling for cases of overlapping subnet ranges, but it seems more straightforward for Docker to handle this (as it already does on the default subnet it seems to hardcode).

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Apr 5, 2016

Contributor

--bip only affects the docker0 bridge, which is for the default network only.

Contributor

cpuguy83 commented Apr 5, 2016

--bip only affects the docker0 bridge, which is for the default network only.

@dimaspivak

This comment has been minimized.

Show comment
Hide comment
@dimaspivak

dimaspivak Apr 5, 2016

Yeah, that's what my experimentation led to me to understanding. Perhaps this is more of a feature request, in that case, to allow specification of a default subnet from which to start looking for open ones when creating new bridge networks?

dimaspivak commented Apr 5, 2016

Yeah, that's what my experimentation led to me to understanding. Perhaps this is more of a feature request, in that case, to allow specification of a default subnet from which to start looking for open ones when creating new bridge networks?

@holgerreif

This comment has been minimized.

Show comment
Hide comment
@holgerreif

holgerreif Apr 6, 2016

This is a feature request and I support it!

docker daemon should have two additional options:

  • --net-cdir specifies an ip range from which docker network create should get their addresses from. This is similiar to the --bip option for traditional docker0 bridge w/o specifying the IP for the dockerhost on the bridge.
  • --net-size spedifies the size of a docker network created network via its subnet len.
    Thus we can have automatic adress assigment w/o clashes with existing networks.

Currently we have to manage ip subnets for networks manually, which is especially ugly when using docker-compose. The reason is basically the same as mentioned in #21329: not keeping track of networks.

Example:

--net-cdir=10.1.0.0/8 --net-size=24 would select 10.1.0.0/24 for the first network, 10.1.1.0/24 for the second one and so on. Conflicts with existing networks should be checked as usual, overlapping with --bip Parameter should be allowed.

Instead of --net-cdir an alternative name for this option would be --net-pool.

holgerreif commented Apr 6, 2016

This is a feature request and I support it!

docker daemon should have two additional options:

  • --net-cdir specifies an ip range from which docker network create should get their addresses from. This is similiar to the --bip option for traditional docker0 bridge w/o specifying the IP for the dockerhost on the bridge.
  • --net-size spedifies the size of a docker network created network via its subnet len.
    Thus we can have automatic adress assigment w/o clashes with existing networks.

Currently we have to manage ip subnets for networks manually, which is especially ugly when using docker-compose. The reason is basically the same as mentioned in #21329: not keeping track of networks.

Example:

--net-cdir=10.1.0.0/8 --net-size=24 would select 10.1.0.0/24 for the first network, 10.1.1.0/24 for the second one and so on. Conflicts with existing networks should be checked as usual, overlapping with --bip Parameter should be allowed.

Instead of --net-cdir an alternative name for this option would be --net-pool.

@wenchma

This comment has been minimized.

Show comment
Hide comment
@wenchma

wenchma Apr 7, 2016

Contributor

@holgerreif --net-cidr and --net-size two additaional options will be welcomed to specify the subnet pool, I also need this feature.

Contributor

wenchma commented Apr 7, 2016

@holgerreif --net-cidr and --net-size two additaional options will be welcomed to specify the subnet pool, I also need this feature.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Apr 7, 2016

Contributor

@dimaspivak @wenchma @holgerreif As you might know already docker network create already supports --subnet & --ip-range options just for this purpose. What is the purpose of adding more flags to the daemon just for the sake of defining pre-defined pools (for automatic assignment ? ).

Contributor

mavenugo commented Apr 7, 2016

@dimaspivak @wenchma @holgerreif As you might know already docker network create already supports --subnet & --ip-range options just for this purpose. What is the purpose of adding more flags to the daemon just for the sake of defining pre-defined pools (for automatic assignment ? ).

@dimaspivak

This comment has been minimized.

Show comment
Hide comment
@dimaspivak

dimaspivak Apr 7, 2016

Exactly, I see this as a daemon option (or, if such a thing exists, as a host-level option for the bridge network driver). In my case, I create a half-dozen bridge networks simultaneously. If I leave out --subnet when starting each, I have the desired behavior where they pick their own subnet, but this seems hard-coded to start looking in the 172... range that conflicts with other hosts on my local network. If I include --subnet, I end up having to build a not-insignificant amount of exception handling to recognize when the client complains about the requested subnet overlapping with an existing one, at which time it then has to try again (potentially hitting other overlaps along the way) until it gets an available one. Essentially, I'm saying that Docker already supports the correct behavior, but simply seems to have hardcoded the IP range to use when starting its search for free pools.

dimaspivak commented Apr 7, 2016

Exactly, I see this as a daemon option (or, if such a thing exists, as a host-level option for the bridge network driver). In my case, I create a half-dozen bridge networks simultaneously. If I leave out --subnet when starting each, I have the desired behavior where they pick their own subnet, but this seems hard-coded to start looking in the 172... range that conflicts with other hosts on my local network. If I include --subnet, I end up having to build a not-insignificant amount of exception handling to recognize when the client complains about the requested subnet overlapping with an existing one, at which time it then has to try again (potentially hitting other overlaps along the way) until it gets an available one. Essentially, I'm saying that Docker already supports the correct behavior, but simply seems to have hardcoded the IP range to use when starting its search for free pools.

@holgerreif

This comment has been minimized.

Show comment
Hide comment
@holgerreif

holgerreif Apr 7, 2016

Actually it's a PITA to manage/select subnets manually.
OTOH I usually do not care about the actual ip addrs choosen, as long as they stick to a defined network range not conflicting with the rest of the network far outside docker host.

BTW: I know the (note from userguide)[https://docs.docker.com/engine/userguide/networking/work-with-networks/#create-networks] It is highly recommended to use the --subnet option while creating a network. But defining a single range to take networks from once is much more pleasure than to do the selection every time you actually create a network (or let docker-compose create one for you).

As @dimaspivak already stated: the logic to pick the right subnet is already there! You just need to feed the right params (starting point --net-pool and size --net-size)

As a goodie (I guess this is just feeding the network select logic with specific parameter to override the global option) one can image to add the parameter --net-size to docker network create as well. And as an option of docker-compose file too.
Well, this is another, more complex issue, since this goodie would require to change the docker remote API...

holgerreif commented Apr 7, 2016

Actually it's a PITA to manage/select subnets manually.
OTOH I usually do not care about the actual ip addrs choosen, as long as they stick to a defined network range not conflicting with the rest of the network far outside docker host.

BTW: I know the (note from userguide)[https://docs.docker.com/engine/userguide/networking/work-with-networks/#create-networks] It is highly recommended to use the --subnet option while creating a network. But defining a single range to take networks from once is much more pleasure than to do the selection every time you actually create a network (or let docker-compose create one for you).

As @dimaspivak already stated: the logic to pick the right subnet is already there! You just need to feed the right params (starting point --net-pool and size --net-size)

As a goodie (I guess this is just feeding the network select logic with specific parameter to override the global option) one can image to add the parameter --net-size to docker network create as well. And as an option of docker-compose file too.
Well, this is another, more complex issue, since this goodie would require to change the docker remote API...

@LukasBacigal

This comment has been minimized.

Show comment
Hide comment
@LukasBacigal

LukasBacigal May 2, 2016

I guess we are trying to modify those pools.
docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go
it is not uncommon to use 172.17-31.x.x ... in company environment, and I faced problem of overlapping earlier.
so +1 for daemon feature to modify default pool.

LukasBacigal commented May 2, 2016

I guess we are trying to modify those pools.
docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go
it is not uncommon to use 172.17-31.x.x ... in company environment, and I faced problem of overlapping earlier.
so +1 for daemon feature to modify default pool.

@jameseck

This comment has been minimized.

Show comment
Hide comment
@jameseck

jameseck May 24, 2016

I would also like to +1 for this feature. I'm in the same situation where new networks created with "docker network create" or docker-compose, etc still use 172.16.x.x and this address range is already in use by other internal services that I need to access.

jameseck commented May 24, 2016

I would also like to +1 for this feature. I'm in the same situation where new networks created with "docker network create" or docker-compose, etc still use 172.16.x.x and this address range is already in use by other internal services that I need to access.

@nschoe

This comment has been minimized.

Show comment
Hide comment
@nschoe

nschoe May 25, 2016

I'll +1 this too, as explained in the issue mentioned above, I think we need:

  1. to be able to specify the subnet
  2. to be able to specify the prefix-length (in my case, I don't really care if this is in 172.x, but I do care that it creates /16 subnets by default as I'm greatly limited, and I'd prefer to use /24)

So +1 ^^

nschoe commented May 25, 2016

I'll +1 this too, as explained in the issue mentioned above, I think we need:

  1. to be able to specify the subnet
  2. to be able to specify the prefix-length (in my case, I don't really care if this is in 172.x, but I do care that it creates /16 subnets by default as I'm greatly limited, and I'd prefer to use /24)

So +1 ^^

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 27, 2016

Contributor

@aboch can you PTAL

Contributor

mavenugo commented May 27, 2016

@aboch can you PTAL

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo May 28, 2016

Contributor

Suggestion: as a workaround, if you want to prevent Docker from using specific subnets, you can do:

docker network create --subnet 172.16.0.0/24 reserved-172-16-0-0-24

Then that subnet won't be used by Docker when auto-allocating address space to other networks.

Contributor

jpetazzo commented May 28, 2016

Suggestion: as a workaround, if you want to prevent Docker from using specific subnets, you can do:

docker network create --subnet 172.16.0.0/24 reserved-172-16-0-0-24

Then that subnet won't be used by Docker when auto-allocating address space to other networks.

@codablock

This comment has been minimized.

Show comment
Hide comment
@codablock

codablock Jun 1, 2016

We have the same problem with overlapping networks, causing random network failures on our CI servers. We also can not use the --subnet option from "docker network create" due to the custom clash avoidance which would be needed.

@jpetazzo Did you test your workaround? Wouldn't it still cause routing failures due to duplicate/clashing routes?

codablock commented Jun 1, 2016

We have the same problem with overlapping networks, causing random network failures on our CI servers. We also can not use the --subnet option from "docker network create" due to the custom clash avoidance which would be needed.

@jpetazzo Did you test your workaround? Wouldn't it still cause routing failures due to duplicate/clashing routes?

@shane-axiom

This comment has been minimized.

Show comment
Hide comment
@shane-axiom

shane-axiom Jun 3, 2016

+1, badly needed! docker network create is creating networks which override active subnets in our physical network.

shane-axiom commented Jun 3, 2016

+1, badly needed! docker network create is creating networks which override active subnets in our physical network.

@shane-axiom

This comment has been minimized.

Show comment
Hide comment
@shane-axiom

shane-axiom Jun 3, 2016

@jpetazzo @codablock I tried the suggested workaround, but as @codablock mentioned this still causes routing failures with existing non-docker subnets. If you're just trying to blacklist an arbitrary subnet from being used by docker network create this will work (because it'll already be in use by another docker network), but if your problem is docker creating subnets that conflict with existing subnets on your network this isn't a viable workaround.

shane-axiom commented Jun 3, 2016

@jpetazzo @codablock I tried the suggested workaround, but as @codablock mentioned this still causes routing failures with existing non-docker subnets. If you're just trying to blacklist an arbitrary subnet from being used by docker network create this will work (because it'll already be in use by another docker network), but if your problem is docker creating subnets that conflict with existing subnets on your network this isn't a viable workaround.

@codablock

This comment has been minimized.

Show comment
Hide comment
@codablock

codablock Jun 6, 2016

Can anyone of the Docker team give an estimate if a fix (or a new feature) for this could be implemented for 1.12 or 1.13? Depending on this we could decide how much effort to put into a workaround.

codablock commented Jun 6, 2016

Can anyone of the Docker team give an estimate if a fix (or a new feature) for this could be implemented for 1.12 or 1.13? Depending on this we could decide how much effort to put into a workaround.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jun 6, 2016

Contributor

@codablock The freeze for features for 1.12 is June 14, so it is going to be tight to get a fix in by then, and I am not aware of anyone working on it. Of course if anyone wants to we will make best efforts to get it in for 1.12, as it is obviously a serious issue for some users.

Contributor

justincormack commented Jun 6, 2016

@codablock The freeze for features for 1.12 is June 14, so it is going to be tight to get a fix in by then, and I am not aware of anyone working on it. Of course if anyone wants to we will make best efforts to get it in for 1.12, as it is obviously a serious issue for some users.

@Kodiack

This comment has been minimized.

Show comment
Hide comment
@Kodiack

Kodiack Jul 5, 2016

@martynsmith found a nice workaround today after reading some of Docker's source code. Docker will not create networks that conflict with nameserver entries in /etc/resolv.conf, so adding a bogus entry there can prevent Docker from creating networks in a subnet that may already be used for something else.

In our case, the 172.17.0.0/16 subnet would conflict with a VPN, so people could not connect to their systems or other internal services if Docker was installed on a system with the default network in use. We were able to change the docker0 range with the --bip argument in /etc/default/docker, but creating new Docker networks (e.g. with Compose) could cause conflicts to arise again.

The fairly simple workaround in our case involved adding this line to /etc/resolv.conf:
nameserver 172.17.254.254

With that line, Docker no longer creates new networks in the 172.17.0.0/16 subnet, saving us a good amount of headache when we haven't specifically defined how to set up a Docker network. Though it's also worth noting that Docker states that it's best to manually define a network (e.g. in a compose file), which is what I've been doing to this point (and will continue to do):

https://docs.docker.com/engine/userguide/networking/work-with-networks/#/create-networks

"Note : It is highly recommended to use the --subnet option while creating a network. If the --subnet is not specified, the docker daemon automatically chooses and assigns a subnet for the network and it could overlap with another subnet in your infrastructure that is not managed by docker. Such overlaps can cause connectivity issues or failures when containers are connected to that network."

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

Kodiack commented Jul 5, 2016

@martynsmith found a nice workaround today after reading some of Docker's source code. Docker will not create networks that conflict with nameserver entries in /etc/resolv.conf, so adding a bogus entry there can prevent Docker from creating networks in a subnet that may already be used for something else.

In our case, the 172.17.0.0/16 subnet would conflict with a VPN, so people could not connect to their systems or other internal services if Docker was installed on a system with the default network in use. We were able to change the docker0 range with the --bip argument in /etc/default/docker, but creating new Docker networks (e.g. with Compose) could cause conflicts to arise again.

The fairly simple workaround in our case involved adding this line to /etc/resolv.conf:
nameserver 172.17.254.254

With that line, Docker no longer creates new networks in the 172.17.0.0/16 subnet, saving us a good amount of headache when we haven't specifically defined how to set up a Docker network. Though it's also worth noting that Docker states that it's best to manually define a network (e.g. in a compose file), which is what I've been doing to this point (and will continue to do):

https://docs.docker.com/engine/userguide/networking/work-with-networks/#/create-networks

"Note : It is highly recommended to use the --subnet option while creating a network. If the --subnet is not specified, the docker daemon automatically chooses and assigns a subnet for the network and it could overlap with another subnet in your infrastructure that is not managed by docker. Such overlaps can cause connectivity issues or failures when containers are connected to that network."

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

@LukasBacigal

This comment has been minimized.

Show comment
Hide comment
@LukasBacigal

LukasBacigal Jul 14, 2016

Should we open another issue as feature request? or change labels to feature and unlabel v 1.10

LukasBacigal commented Jul 14, 2016

Should we open another issue as feature request? or change labels to feature and unlabel v 1.10

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 14, 2016

Contributor

(version) Labels are automatically set based on the version supplied in the original issue, no need to change.

Contributor

cpuguy83 commented Jul 14, 2016

(version) Labels are automatically set based on the version supplied in the original issue, no need to change.

@creynders

This comment has been minimized.

Show comment
Hide comment
@creynders

creynders commented Sep 21, 2016

👍 💯

@panga

This comment has been minimized.

Show comment
Hide comment
@panga

panga Oct 3, 2016

I'm having the same issue with a corporate network, please add an option to change default subnets (+1)

panga commented Oct 3, 2016

I'm having the same issue with a corporate network, please add an option to change default subnets (+1)

@sabretus

This comment has been minimized.

Show comment
Hide comment
@sabretus

sabretus Oct 19, 2016

Same here, we use docker to run tests and it's very useful when subnets are allocated automatically, but their size is too big so we can't start more than 15 parallel tests (172.17. - 172.31.) at the same time.

sabretus commented Oct 19, 2016

Same here, we use docker to run tests and it's very useful when subnets are allocated automatically, but their size is too big so we can't start more than 15 parallel tests (172.17. - 172.31.) at the same time.

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch Oct 26, 2016

Contributor

@dimaspivak

It seems consensus was reached around the feature request in #21776 (comment) by @holgerreif.

This request has come in other occasions via other issues.

I think we should change this issue's description to reflect that, so other docker users can quickly find it.

Contributor

aboch commented Oct 26, 2016

@dimaspivak

It seems consensus was reached around the feature request in #21776 (comment) by @holgerreif.

This request has come in other occasions via other issues.

I think we should change this issue's description to reflect that, so other docker users can quickly find it.

@dimaspivak dimaspivak changed the title from Docker network creation doesn't respect bridge IP for subnets to Add daemon options for Docker network subnets Oct 27, 2016

@dimaspivak

This comment has been minimized.

Show comment
Hide comment
@dimaspivak

dimaspivak Oct 27, 2016

How's that look to you, @aboch?

dimaspivak commented Oct 27, 2016

How's that look to you, @aboch?

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch Oct 28, 2016

Contributor

looks good. thanks.

Contributor

aboch commented Oct 28, 2016

looks good. thanks.

@twang2218

This comment has been minimized.

Show comment
Hide comment
@twang2218

twang2218 Dec 28, 2016

I believe we do need an option to configure the default PredefinedBroadNetworks, or at least can option to exclude a subnet list for the allocation. As in Docker for Mac/Windows, the /etc/resolv.conf or routing table trick will not works, as the Linux host is running in a VM, cannot read physical host's /etc/resolv.conf or routing table.

twang2218 commented Dec 28, 2016

I believe we do need an option to configure the default PredefinedBroadNetworks, or at least can option to exclude a subnet list for the allocation. As in Docker for Mac/Windows, the /etc/resolv.conf or routing table trick will not works, as the Linux host is running in a VM, cannot read physical host's /etc/resolv.conf or routing table.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 28, 2016

Member

@twang2218 would #29376 solve your use case?

Member

thaJeztah commented Dec 28, 2016

@twang2218 would #29376 solve your use case?

@twang2218

This comment has been minimized.

Show comment
Hide comment
@twang2218

twang2218 Dec 28, 2016

@thaJeztah Yes, as the default address pool can be set through the daemon.json:

{"default-address-pools":[{"scope":"local","base":"172.80.0.0/16","size":24},{"scope":"global","base":"172.90.0.0/16","size":24}]}

the subnet address range overlapping problem can be avoided in Docker for Mac/Windows, so I think my problem can be solved by #29376 .

twang2218 commented Dec 28, 2016

@thaJeztah Yes, as the default address pool can be set through the daemon.json:

{"default-address-pools":[{"scope":"local","base":"172.80.0.0/16","size":24},{"scope":"global","base":"172.90.0.0/16","size":24}]}

the subnet address range overlapping problem can be avoided in Docker for Mac/Windows, so I think my problem can be solved by #29376 .

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 28, 2016

Member

perfect, thanks!

Member

thaJeztah commented Dec 28, 2016

perfect, thanks!

@jmkgreen

This comment has been minimized.

Show comment
Hide comment
@jmkgreen

jmkgreen Jun 8, 2017

Is there any movement here? The discussions seem to have moved to the pull request #29376 where it was discovered that deployment in a Swarm environment will not necessarily honour that new behaviour.

jmkgreen commented Jun 8, 2017

Is there any movement here? The discussions seem to have moved to the pull request #29376 where it was discovered that deployment in a Swarm environment will not necessarily honour that new behaviour.

@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 7, 2017

I tried the /etc/resolv.conf tech. It did not work.
Is there anyway to tell Docker not to hand out a certain address within the subnet mask?
In my situation I have a bridge to an external net. I need to talk to one external machine at 172.27.153.1.
( 172.27.153.0/24 ) The very first address Docker hands out when starting a new container is 172.27.153.1. I also gives this address to the the host bridge as well. So I can no longer talk to the external 172.27.153.1.
I am stuck. Any suggestions?

robertknox commented Jul 7, 2017

I tried the /etc/resolv.conf tech. It did not work.
Is there anyway to tell Docker not to hand out a certain address within the subnet mask?
In my situation I have a bridge to an external net. I need to talk to one external machine at 172.27.153.1.
( 172.27.153.0/24 ) The very first address Docker hands out when starting a new container is 172.27.153.1. I also gives this address to the the host bridge as well. So I can no longer talk to the external 172.27.153.1.
I am stuck. Any suggestions?

@jmkgreen

This comment has been minimized.

Show comment
Hide comment
@jmkgreen

jmkgreen Jul 7, 2017

Note: We're using Docker 17.06.ce with Swarm.

Please see https://docs.docker.com/engine/userguide/networking/get-started-overlay/#create-the-overlay-network which includes a really important recommendation to always specify the desired subnet of your new network.

We think Docker's position is that if you have non-Docker private subnets, you absolutely must create subnets for Docker's own use manually and configure them all to not conflict.

Also be advised that in our experience, when using the overlay network driver, Docker will consume IPs for it's own use in addition to the ones your declared containers will use. This has caused us to calculate the subnet size based on the expected number of containers then double it, else stacks will not start fully and they provide no feedback as to why.

jmkgreen commented Jul 7, 2017

Note: We're using Docker 17.06.ce with Swarm.

Please see https://docs.docker.com/engine/userguide/networking/get-started-overlay/#create-the-overlay-network which includes a really important recommendation to always specify the desired subnet of your new network.

We think Docker's position is that if you have non-Docker private subnets, you absolutely must create subnets for Docker's own use manually and configure them all to not conflict.

Also be advised that in our experience, when using the overlay network driver, Docker will consume IPs for it's own use in addition to the ones your declared containers will use. This has caused us to calculate the subnet size based on the expected number of containers then double it, else stacks will not start fully and they provide no feedback as to why.

@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 7, 2017

Thanks very much for the response.

I am creating a subnet for Docker. The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.) If I pass docker a subnet of 172.27.153.0/24, the very first container created will get 172.27.153.1.

I am not using the overlay driver with swarm. I am bridging to a linux bridge on the host. This allows me access to a subnet (external) and the all important 172.27.153.1 box.

I have tried putting in 172.27.153.1 into /etc/resolv.conf on the host machine as described above to force docker not to use that ip when handing out ips -- no luck. Is there another way to do this?

I am trying to think of options: I have read that I can pass into docker "run" an ip address:"docker run --ip XXXXXX". Maybe that is my option? I could manage all the ips myself, handing out an ip I know is safe at "docker run time"? I need to create 200 docker container's each acting as a certain type of device that will interact with the 172.27.153.1 box that sits on the network.

I very much appreciate your help on this issue.

robertknox commented Jul 7, 2017

Thanks very much for the response.

I am creating a subnet for Docker. The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.) If I pass docker a subnet of 172.27.153.0/24, the very first container created will get 172.27.153.1.

I am not using the overlay driver with swarm. I am bridging to a linux bridge on the host. This allows me access to a subnet (external) and the all important 172.27.153.1 box.

I have tried putting in 172.27.153.1 into /etc/resolv.conf on the host machine as described above to force docker not to use that ip when handing out ips -- no luck. Is there another way to do this?

I am trying to think of options: I have read that I can pass into docker "run" an ip address:"docker run --ip XXXXXX". Maybe that is my option? I could manage all the ips myself, handing out an ip I know is safe at "docker run time"? I need to create 200 docker container's each acting as a certain type of device that will interact with the 172.27.153.1 box that sits on the network.

I very much appreciate your help on this issue.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 7, 2017

Contributor

@robertknox You can specify the range of IP's to use beyond just the subnet. On network create it's --ip-range, for the default bridge you set it using --fixed-cidr in the daemon options.

Contributor

cpuguy83 commented Jul 7, 2017

@robertknox You can specify the range of IP's to use beyond just the subnet. On network create it's --ip-range, for the default bridge you set it using --fixed-cidr in the daemon options.

@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 7, 2017

thanks very much for your suggestion, but I am not using the default bridge. I am using a custom bridge to a linux bridge to a particular port on my machine. I am passing in the --ip-range option, but it gives me just that -- a range. I cannot say "don't use this one ip address."

docker network create --driver=bridge --ip-range=172.27.153.0/24 --subnet=172.27.153.0/24 -o "com.docker.network.bridge.name=br0" br0

The first container created will get the ip = 172.27.153.1 and that conflicts with another box on this network.

Thanks again.

robertknox commented Jul 7, 2017

thanks very much for your suggestion, but I am not using the default bridge. I am using a custom bridge to a linux bridge to a particular port on my machine. I am passing in the --ip-range option, but it gives me just that -- a range. I cannot say "don't use this one ip address."

docker network create --driver=bridge --ip-range=172.27.153.0/24 --subnet=172.27.153.0/24 -o "com.docker.network.bridge.name=br0" br0

The first container created will get the ip = 172.27.153.1 and that conflicts with another box on this network.

Thanks again.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jul 7, 2017

Contributor
Contributor

cpuguy83 commented Jul 7, 2017

@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 7, 2017

I didn't know --ip-range can be used multiple times to indicate the ip nums slice desired.
thanks so much

robertknox commented Jul 7, 2017

I didn't know --ip-range can be used multiple times to indicate the ip nums slice desired.
thanks so much

@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 7, 2017

Could you give an example?

robertknox commented Jul 7, 2017

Could you give an example?

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch Jul 8, 2017

Contributor

@robertknox

The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.)

--aux-address is the option to tell IPAM which addresses are already taken in your network. So they won't be given out to containers.

Example:

docker network create --subnet 12.12.0.0/24 --aux-address dns=12.12.0.53 --aux-address dut=12.12.0.72 --gateway 12.12.0.254 my_phy_network
Contributor

aboch commented Jul 8, 2017

@robertknox

The problem is that within that subnet there are certain IPs that are already taken ( In my case, 172.27.153.1 -- the gateway, dns and DUT machine.)

--aux-address is the option to tell IPAM which addresses are already taken in your network. So they won't be given out to containers.

Example:

docker network create --subnet 12.12.0.0/24 --aux-address dns=12.12.0.53 --aux-address dut=12.12.0.72 --gateway 12.12.0.254 my_phy_network
@robertknox

This comment has been minimized.

Show comment
Hide comment
@robertknox

robertknox Jul 8, 2017

Thanks very much

robertknox commented Jul 8, 2017

Thanks very much

@archenroot

This comment has been minimized.

Show comment
Hide comment
@archenroot

archenroot Aug 9, 2017

I got it working on Centos 7.3 with Docker 17.06.0 CE with just small issues:
#34461

archenroot commented Aug 9, 2017

I got it working on Centos 7.3 with Docker 17.06.0 CE with just small issues:
#34461

@andypost

This comment has been minimized.

Show comment
Hide comment
@andypost

andypost Sep 16, 2017

btw Why not make subnets smaller? for example using mask 8 we can get *8 subnets 172.19.0.0/8

andypost commented Sep 16, 2017

btw Why not make subnets smaller? for example using mask 8 we can get *8 subnets 172.19.0.0/8

@zarbis

This comment has been minimized.

Show comment
Hide comment
@zarbis

zarbis Sep 16, 2017

@andypost How exactly will it solve problem of subnet shortage?

zarbis commented Sep 16, 2017

@andypost How exactly will it solve problem of subnet shortage?

@rdxmb

This comment has been minimized.

Show comment
Hide comment
@rdxmb

rdxmb Oct 4, 2017

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

This seems to be the best solution for me.
And instead of defining subnets via daemon-cli, this could (and should) be done via config-File in /etc/docker like https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/

rdxmb commented Oct 4, 2017

For simplicity, it really would be nice to be able to explicitly blacklist specific subnets from being used.

This seems to be the best solution for me.
And instead of defining subnets via daemon-cli, this could (and should) be done via config-File in /etc/docker like https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/

@batmat

This comment has been minimized.

Show comment
Hide comment
@batmat

batmat Jan 3, 2018

The workaround with resolv.conf above did not work for me using Docker 17.12.0-ce. The only thing that did was using @jpetazzo's workaround + deleting the associated route, which is quite hackish. Also because this fake network will get wiped out each time one runs docker network prune...

TL;DR (if I want to prevent 172.x.x.x usage, which includes especially some of our VPN range):

docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0

Illustrated:

$ docker network create network$RANDOM; docker network create network$RANDOM
605ad9cd3ef6f1572d421b6de3d730af594cbd26b90c4471100921e9c00221dd
9f453f5b28d2a384b73f5d259e2dc366da9d59f94a60776e6ba091287a648a97
$ docker network ls                                                         
NETWORK ID          NAME                DRIVER              SCOPE
3b34008790f1        bridge              bridge              local
67b97edbee5e        host                host                local
9f453f5b28d2        network16470        bridge              local
605ad9cd3ef6        network7807         bridge              local
bd0efb23366a        none                null                local
$ ifconfig | grep br- -A1                                                   
br-605ad9cd3ef6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
--
br-9f453f5b28d2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
$ docker network prune     # above you see the 172.18.0.1 clash I do not want
WARNING! This will remove all networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
network7807
network16470

$ docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
63c91b810977cd883a8f03e93f43c5c55c5dd3b040aeda18623895ce657fe0ae
$ route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0
SIOCDELRT: Operation not permitted
$ sudo route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0                     7
$ docker network create network$RANDOM; docker network create network$RANDOM
7754974a0a2c853a83913a760f797e504fbc9b8ca3fe015d1b7862d9f602a2ad
2b93f62df1470d8ae36a981a24b3ebea9308172364fa0e465c0ac948fd71856d
$ ifconfig | grep br- -A1
br-2b93f62df147: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.32.1  netmask 255.255.240.0  broadcast 192.168.47.255
--
br-63c91b810977: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.0.0.1  netmask 255.0.0.0  broadcast 172.255.255.255
--
br-7754974a0a2c: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.16.1  netmask 255.255.240.0  broadcast 192.168.31.255

Also: FTR, the solution involving using explicit parameters does not work for my case as I'm using docker-compose. So I could probably modify every docker-compose.yml I'm using out there, but I suspect that would slightly unideal :P.

I'd welcome any hint for a less-hackish/cleaner solution or workaround. Thank you everyone and thanks again for that great tool that is Docker!

batmat commented Jan 3, 2018

The workaround with resolv.conf above did not work for me using Docker 17.12.0-ce. The only thing that did was using @jpetazzo's workaround + deleting the associated route, which is quite hackish. Also because this fake network will get wiped out each time one runs docker network prune...

TL;DR (if I want to prevent 172.x.x.x usage, which includes especially some of our VPN range):

docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0

Illustrated:

$ docker network create network$RANDOM; docker network create network$RANDOM
605ad9cd3ef6f1572d421b6de3d730af594cbd26b90c4471100921e9c00221dd
9f453f5b28d2a384b73f5d259e2dc366da9d59f94a60776e6ba091287a648a97
$ docker network ls                                                         
NETWORK ID          NAME                DRIVER              SCOPE
3b34008790f1        bridge              bridge              local
67b97edbee5e        host                host                local
9f453f5b28d2        network16470        bridge              local
605ad9cd3ef6        network7807         bridge              local
bd0efb23366a        none                null                local
$ ifconfig | grep br- -A1                                                   
br-605ad9cd3ef6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
--
br-9f453f5b28d2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
$ docker network prune     # above you see the 172.18.0.1 clash I do not want
WARNING! This will remove all networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
network7807
network16470

$ docker network create --subnet 172.0.0.0/8 reserved-172-0-0-0-8
63c91b810977cd883a8f03e93f43c5c55c5dd3b040aeda18623895ce657fe0ae
$ route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0
SIOCDELRT: Operation not permitted
$ sudo route del -net 172.0.0.0 gw 0.0.0.0 netmask 255.0.0.0                     7
$ docker network create network$RANDOM; docker network create network$RANDOM
7754974a0a2c853a83913a760f797e504fbc9b8ca3fe015d1b7862d9f602a2ad
2b93f62df1470d8ae36a981a24b3ebea9308172364fa0e465c0ac948fd71856d
$ ifconfig | grep br- -A1
br-2b93f62df147: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.32.1  netmask 255.255.240.0  broadcast 192.168.47.255
--
br-63c91b810977: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.0.0.1  netmask 255.0.0.0  broadcast 172.255.255.255
--
br-7754974a0a2c: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.16.1  netmask 255.255.240.0  broadcast 192.168.31.255

Also: FTR, the solution involving using explicit parameters does not work for my case as I'm using docker-compose. So I could probably modify every docker-compose.yml I'm using out there, but I suspect that would slightly unideal :P.

I'd welcome any hint for a less-hackish/cleaner solution or workaround. Thank you everyone and thanks again for that great tool that is Docker!

@moisei

This comment has been minimized.

Show comment
Hide comment

moisei commented Jan 16, 2018

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jan 16, 2018

Member

@moisei Pull-request #29376 was closed, not merged, so that feature is not available (yet) in Docker

Member

thaJeztah commented Jan 16, 2018

@moisei Pull-request #29376 was closed, not merged, so that feature is not available (yet) in Docker

@orodbhen

This comment has been minimized.

Show comment
Hide comment
@orodbhen

orodbhen Jan 18, 2018

This would be hugely useful. I'm dealing with messaging software that requires all member host IPs to be recorded in a configuration file. The simplest way I can find of doing this is creating the network, querying Docker for the subnet, and dumping all 4096 possible IPs into the file. That's likely going to create performance issues for the messaging software, and a lot of extra traffic.

I can't know what the container IPs will be until they've started up, so this appears to be the only way.

If I set the subnet explicitly, then I have to code up all the logic to prevent overlap.

It would be so much better to be able to set the subnet size in the compose file.

orodbhen commented Jan 18, 2018

This would be hugely useful. I'm dealing with messaging software that requires all member host IPs to be recorded in a configuration file. The simplest way I can find of doing this is creating the network, querying Docker for the subnet, and dumping all 4096 possible IPs into the file. That's likely going to create performance issues for the messaging software, and a lot of extra traffic.

I can't know what the container IPs will be until they've started up, so this appears to be the only way.

If I set the subnet explicitly, then I have to code up all the logic to prevent overlap.

It would be so much better to be able to set the subnet size in the compose file.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jan 19, 2018

Member

It would be so much better to be able to set the subnet size in the compose file.

You can already set network options (such as subnet) in the compose-file; https://docs.docker.com/compose/compose-file/#network-configuration-reference

The feature requested here is to set a default on the daemon, that's used if you did not specify those options when creating a network

Member

thaJeztah commented Jan 19, 2018

It would be so much better to be able to set the subnet size in the compose file.

You can already set network options (such as subnet) in the compose-file; https://docs.docker.com/compose/compose-file/#network-configuration-reference

The feature requested here is to set a default on the daemon, that's used if you did not specify those options when creating a network

@orodbhen

This comment has been minimized.

Show comment
Hide comment
@orodbhen

orodbhen Jan 19, 2018

@thaJeztah There's also some discussion regarding making the option available from the
client side as well. Either or both would be very helpful.

You can set the subnet explicitly in the compose file, but there's no
option to simply set the size and let Docker select the address.

Regardless of how it's done, the idea is to avoid having to validate the
user-provided subnet to prevent overlap.

Incidentally, though, there is an easy way to do this in Python with the
ipaddress module.

net1 = ip_address(subnet1)
net2 = ip_address('subnet2)
invalid = net1.overlaps(net2)

orodbhen commented Jan 19, 2018

@thaJeztah There's also some discussion regarding making the option available from the
client side as well. Either or both would be very helpful.

You can set the subnet explicitly in the compose file, but there's no
option to simply set the size and let Docker select the address.

Regardless of how it's done, the idea is to avoid having to validate the
user-provided subnet to prevent overlap.

Incidentally, though, there is an easy way to do this in Python with the
ipaddress module.

net1 = ip_address(subnet1)
net2 = ip_address('subnet2)
invalid = net1.overlaps(net2)
@cristianocasella

This comment has been minimized.

Show comment
Hide comment
@cristianocasella

cristianocasella Feb 8, 2018

hi guys, I put a PR for this trouble: docker/libnetwork#2072

cristianocasella commented Feb 8, 2018

hi guys, I put a PR for this trouble: docker/libnetwork#2072

@marcoreni

This comment has been minimized.

Show comment
Hide comment
@marcoreni

marcoreni Mar 22, 2018

Hello @cristianocasella , looks like a PR ( #36396 ) is almost ready for merge and should allow the user to specify one or more address pool, thus guaranteeing more freedom.

marcoreni commented Mar 22, 2018

Hello @cristianocasella , looks like a PR ( #36396 ) is almost ready for merge and should allow the user to specify one or more address pool, thus guaranteeing more freedom.

@Yajo

This comment has been minimized.

Show comment
Hide comment
@Yajo

Yajo May 29, 2018

#36396 is merged, shouldn't this be closed?

Yajo commented May 29, 2018

#36396 is merged, shouldn't this be closed?

@cristianocasella

This comment has been minimized.

Show comment
Hide comment
@cristianocasella

cristianocasella May 29, 2018

Thanks for your support. With #36396 this can be closed.

cristianocasella commented May 29, 2018

Thanks for your support. With #36396 this can be closed.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 May 29, 2018

Contributor

Thanks!

Contributor

cpuguy83 commented May 29, 2018

Thanks!

@cpuguy83 cpuguy83 closed this May 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment