Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow user to specify default address pools for docker networks #36396

Merged
merged 1 commit into from May 3, 2018

Conversation

@selansen
Copy link
Contributor

@selansen selansen commented Feb 23, 2018

This is continuation of PR #36054
. I updated all review comments and re-vendor libnetwork. While doing so , I accidentally deleted my commits. Hence I was not able to update into PR 36054.

Description
When user creates a network without specifying a --subnet, docker will pick a subnet for the network from the static set 172.[17-31].0.0/16 and 192.168.[0-240].0/20 for the local scope networks and from the static set 10.[0-255].[0-255].0/24 for the global scope networks.
For different reasons, several users have asked to be able to control these defaults.
This PR brings in a change to allow users to control the default address pools at daemon boot.

As an example,
dockerd --default-address-pools base=10.10.0.0/16,size=24
would allow user to set the 256 pools 10.10.[0-255].0/24 as default for the local scope networks.

Multiple --default-address-pools can be specified.
To specify the option in the config json file:

{"default-address-pools":[{"base":"172.80.0.0/16","size":24},{"base":"172.90.0.0/16","size":24}]}

@selansen selansen requested review from dnephin and vdemeester as code owners Feb 23, 2018
@selansen selansen force-pushed the selansen:master branch 2 times, most recently from d6a1679 to b167edd Feb 24, 2018
@codecov
Copy link

@codecov codecov bot commented Feb 24, 2018

Codecov Report

No coverage uploaded for pull request base (master@d4e48af). Click here to learn what that means.
The diff coverage is 45.45%.

@@            Coverage Diff            @@
##             master   #36396   +/-   ##
=========================================
  Coverage          ?   35.09%           
=========================================
  Files             ?      615           
  Lines             ?    45745           
  Branches          ?        0           
=========================================
  Hits              ?    16053           
  Misses            ?    27582           
  Partials          ?     2110
@selansen
Copy link
Contributor Author

@selansen selansen commented Feb 24, 2018

@thaJeztah , @dnephin , @vdemeester , Pls use this for #36054.
Final Revendor-ing is done. Libnetwork code is merged into upstream with all required changes.

// NetworkConfig stores the daemon-wide networking configurations
type NetworkConfig struct {
// Default address pools for docker networks
DefaultAddressPools []*ipamutils.NetworkToSplit `json:"default-node-local-address-pools,omitempty"`

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

It looks like this is identical in both unix and windows. Can this go into CommonConfig instead?

This comment has been minimized.

@selansen

selansen Feb 26, 2018
Author Contributor

Currently we dont support it on windows.

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

I see you adding this same field in config_windows.go. Should it be removed then?

This comment has been minimized.

@selansen

selansen Feb 26, 2018
Author Contributor

My CI for windows p *windowsRS1 * failed due to build issue. so I had to
modify to fix CI failure.

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

Either way there is a problem here.

Either the code needs to be changed so it's not referenced on windows, or this field needs to be moved into a non-platform specific file. It does not make sense to dupliacte it.

func (p *PoolsOpt) Value() []*types.NetworkToSplit {
var pd []*types.NetworkToSplit
pd = append(pd, *p.values...)
return pd

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

Why not return *p.values ? Might also be good to check for nil.

This comment has been minimized.

@selansen

selansen Feb 26, 2018
Author Contributor

make sense. will modify


// PoolsOpt is a Value type for parsing the default address pools definitions
type PoolsOpt struct {
values *[]*types.NetworkToSplit

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

I don't think this needs to be a pointer. The PoolsOpt receiver is already a pointer, so you should be able to p.values = append(p.values, ...)

//c.Assert(out, checker.Contains, "175.30.0.0/16")
c.Assert(out.IPAM.Config[0].Subnet, checker.Equals, "175.30.0.0/16")

// Create a bridge network and verify its subnet is the second default pool

This comment has been minimized.

@dnephin

dnephin Feb 26, 2018
Member

It looks like these are separate cases? Can you split this into separate Test functions?

This comment has been minimized.

@selansen

selansen Feb 26, 2018
Author Contributor

It is same test case. We are testing it by creating 2 different network and make sure they fall into same user defined subnet. thats all.

@selansen selansen force-pushed the selansen:master branch from b167edd to 0b07fe1 Feb 27, 2018
@selansen
Copy link
Contributor Author

@selansen selansen commented Feb 27, 2018

@dnephin , Addressed all the comments that we discussed yesterday. Pls take a look at it when you have time.

Copy link
Member

@dnephin dnephin left a comment

I guess the tests will fail now. I see why you were using a pointer to a slice.

@@ -15,7 +15,7 @@ type BridgeConfig struct {
// to the docker daemon when you launch it with say: `dockerd -e windows`
type Config struct {
CommonConfig

NetworkConfig

This comment has been minimized.

@dnephin

dnephin Feb 27, 2018
Member

Still not sure why this is being added on windows if you say it isn't used, but at least now the struct isn't duplicated.

This comment has been minimized.

@selansen

selansen Feb 27, 2018
Author Contributor

missed this one. will remove it.

@@ -44,4 +44,6 @@ func installConfigFlags(conf *config.Config, flags *pflag.FlagSet) {
flags.Var(&conf.ShmSize, "default-shm-size", "Default shm size for containers")
flags.BoolVar(&conf.NoNewPrivileges, "no-new-privileges", false, "Set no-new-privileges by default for new containers")
flags.StringVar(&conf.IpcMode, "default-ipc-mode", config.DefaultIpcMode, `Default mode for containers ipc ("shareable" | "private")`)
flags.Var(opts.NewPoolsOpt(conf.NetworkConfig.DefaultAddressPools), "default-node-local-address-pools", "Set the default address pools for node specific local networks")

This comment has been minimized.

@dnephin

dnephin Feb 27, 2018
Member

You can remove "Set the". I see we have it for one other flag, but it reads better without it.

This comment has been minimized.

@selansen

selansen Feb 27, 2018
Author Contributor

corrected it.

func (p *PoolsOpt) String() string {
pools := []string{}
for _, pool := range p.values {
repr := fmt.Sprintf(" %s %s", pool.Base, pool.Size)

This comment has been minimized.

@dnephin

dnephin Feb 27, 2018
Member

Why the leading space in the format? There's already a space added by strings.Join()

This comment has been minimized.

@selansen

selansen Feb 27, 2018
Author Contributor

Corrected it.

// NetworkConfig stores the daemon-wide networking configurations
type NetworkConfig struct {
// Default address pools for docker networks
DefaultAddressPools []*ipamutils.NetworkToSplit `json:"default-node-local-address-pools,omitempty"`

This comment has been minimized.

@dnephin

dnephin Feb 27, 2018
Member

Now I see why you were using a pointer to a slice.

The type here can be PoolsOpt instead of []*ipamutils.NetworkToSplit.

That way you don't need to accept any value in NewPoolsOpt, and to get the value out you can use: DefaultAddressPools.Value()

@thaJeztah thaJeztah added this to backlog in maintainers-session Mar 1, 2018
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Mar 6, 2018

Yup, tests look to be failing;

20:44:12 ----------------------------------------------------------------------
20:44:12 FAIL: docker_api_network_test.go:182: DockerDaemonSuite.TestAPIDaemonNetworkPools
20:44:12 
20:44:12 [d4ac7cc665224] waiting for daemon to start
20:44:12 [d4ac7cc665224] daemon started
20:44:12 
20:44:12 docker_api_network_test.go:199:
20:44:12     //c.Assert(out, checker.Contains, "175.30.0.0/16")
20:44:12     c.Assert(out.IPAM.Config[0].Subnet, checker.Equals, "175.30.0.0/16")
20:44:12 ... obtained string = "172.18.0.0/16"
20:44:12 ... expected string = "175.30.0.0/16"
20:44:12 
20:44:12 [d4ac7cc665224] exiting daemon
@selansen
Copy link
Contributor Author

@selansen selansen commented Mar 6, 2018

@selansen
Copy link
Contributor Author

@selansen selansen commented Mar 14, 2018

@dnephin , All your suggestions have been taken care. Both manual and integration tests are working fine.
@thaJeztah Failures are not due to this changes. I have committed Flaky test issue on separate PR.
Also , libnetwork changes are already merged into moby when you picked up libnetwork for latest fixes.

Copy link
Member

@dnephin dnephin left a comment

Thanks for making these changes

LGTM

@ghost
Copy link

@ghost ghost commented Mar 20, 2018

Looks great! I really can't wait to use it in my setup :)

@vdemeester Could you review this as well?

@selansen
Copy link
Contributor Author

@selansen selansen commented Mar 20, 2018

@vdemeester @thaJeztah , I think we have gone through multiple review cycle on this.
Could pls give final blessing and merge it ?

@ToonSpin
Copy link

@ToonSpin ToonSpin commented Jul 28, 2018

Note that Docker remembers its networks, and this setting is only applied to new networks. If, like me, you are using docker-compose but are still finding routes that do not conform to your settings in the JSON configuration, then executing docker network ls and then docker network rm NETWORK on any offending networks should fix things.

@rca
Copy link

@rca rca commented Aug 6, 2018

@mavenugo in the thread above you mention, "given that you identified my earlier comment on this topic." I'm looking to customize the IPAM for swarm network creation without having to specify the subnet for every stack I deploy. Is there, by chance, some trick that can be used until this is implemented for swarm networks as well.

Thank you!

@selansen
Copy link
Contributor Author

@selansen selansen commented Aug 6, 2018

We are working on overlay network support at this moment. New feature will be available in the next release.
only viable option is using "--subnet" while creating network. Looks like you are using the option already

@rca
Copy link

@rca rca commented Aug 7, 2018

Yep, that's how I've been getting around this, and I'm looking to get out of the subnet management business. ;)

Thanks for the response, @selansen !

@selansen
Copy link
Contributor Author

@selansen selansen commented Aug 7, 2018

Hopefully soon :)

@vincentjorgensen
Copy link

@vincentjorgensen vincentjorgensen commented Jan 29, 2019

I'm using 18.09.1, but I get "unknown flag: --default-address-pools". Do we know which release this will be in? Is there a way I can check so I don't clutter the forum with inane questions? thank you!

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 29, 2019

Should be there in 18.09.1;

dockerd --help

...
   --default-address-pool pool-options       Default address pools for node specific local networks
...

And for swarm;

docker swarm init --help

Usage:	docker swarm init [OPTIONS]

Initialize a swarm

Options:
      --advertise-addr string                  Advertised address (format: <ip|interface>[:port])
      --autolock                               Enable manager autolocking (requiring an unlock key to start a stopped manager)
      --availability string                    Availability of the node ("active"|"pause"|"drain") (default "active")
      --cert-expiry duration                   Validity period for node certificates (ns|us|ms|s|m|h) (default 2160h0m0s)
      --data-path-addr string                  Address or interface to use for data path traffic (format: <ip|interface>)
      --default-addr-pool ipNetSlice           default address pool in CIDR format (default [])
      --default-addr-pool-mask-length uint32   default address pool subnet mask length (default 24)
      --dispatcher-heartbeat duration          Dispatcher heartbeat period (ns|us|ms|s|m|h) (default 5s)
      --external-ca external-ca                Specifications of one or more certificate signing endpoints
      --force-new-cluster                      Force create a new cluster from current state
      --listen-addr node-addr                  Listen address (format: <ip|interface>[:port]) (default 0.0.0.0:2377)
      --max-snapshots uint                     Number of additional Raft snapshots to retain
      --snapshot-interval uint                 Number of log entries between Raft snapshots (default 10000)
      --task-history-limit int                 Task history retention limit (default 5)
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 29, 2019

Ah; I see you're using --default-address-pools (plural); I think that changed to singular during review

@leokun
Copy link

@leokun leokun commented Jan 29, 2019

For docker daemon create a /etc/docker/daemon.json and add :

{
    “bip”: “10.10.0.1/24”
}

Or whatever you need.
Some notice : I had to create a /24 to get /16. But I can’t explain why

@JorisVanEijden
Copy link

@JorisVanEijden JorisVanEijden commented Feb 14, 2019

The option does not seem to be in 18.09.1 for windows:

$ ./dockerd.exe --version
Docker version 18.09.1, build 4c52b90
$ ./dockerd.exe --help

Usage:  dockerd [OPTIONS]

A self-sufficient runtime for containers.

Options:
      --allow-nondistributable-artifacts list   Allow push of
                                                nondistributable
                                                artifacts to registry
      --api-cors-header string                  Set CORS headers in the
                                                Engine API
      --authorization-plugin list               Authorization plugins to load
  -b, --bridge string                           Attach containers to a
                                                virtual switch
      --cluster-advertise string                Address or interface name
                                                to advertise
      --cluster-store string                    URL of the distributed
                                                storage backend
      --cluster-store-opt map                   Set cluster store options
                                                (default map[])
      --config-file string                      Daemon configuration file
      --containerd string                       containerd grpc address
      --cri-containerd                          start containerd with cri
      --data-root string                        Root directory of
                                                persistent Docker state
                                                (default
                                                "C:\\ProgramData\\docker")
  -D, --debug                                   Enable debug mode
      --dns list                                DNS server to use
      --dns-opt list                            DNS options to use
      --dns-search list                         DNS search domains to use
      --exec-opt list                           Runtime execution options
      --exec-root string                        Root directory for
                                                execution state files
                                                (default
                                                "C:\\ProgramData\\docker\\exec-root")
      --experimental                            Enable experimental features
      --fixed-cidr string                       IPv4 subnet for fixed IPs
  -G, --group string                            Users or groups that can
                                                access the named pipe
      --help                                    Print usage
  -H, --host list                               Daemon socket(s) to connect to
      --insecure-registry list                  Enable insecure registry
                                                communication
      --label list                              Set key=value labels to
                                                the daemon
      --log-driver string                       Default driver for
                                                container logs (default
                                                "json-file")
  -l, --log-level string                        Set the logging level
                                                ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --log-opt map                             Default log driver
                                                options for containers
                                                (default map[])
      --max-concurrent-downloads int            Set the max concurrent
                                                downloads for each pull
                                                (default 3)
      --max-concurrent-uploads int              Set the max concurrent
                                                uploads for each push
                                                (default 5)
      --metrics-addr string                     Set default address and
                                                port to serve the metrics
                                                api on
      --mtu int                                 Set the containers network MTU
      --network-control-plane-mtu int           Network Control plane MTU
                                                (default 1500)
      --node-generic-resource list              Advertise user-defined
                                                resource
  -p, --pidfile string                          Path to use for daemon
                                                PID file
      --raw-logs                                Full timestamps without
                                                ANSI coloring
      --register-service                        Register the service and exit
      --registry-mirror list                    Preferred Docker registry
                                                mirror
      --service-name string                     Set the Windows service
                                                name (default "docker")
      --shutdown-timeout int                    Set the default shutdown
                                                timeout (default 15)
      --storage-opt list                        Storage driver options
      --swarm-default-advertise-addr string     Set default address or
                                                interface for swarm
                                                advertised address
      --tls                                     Use TLS; implied by
                                                --tlsverify
      --tlscacert string                        Trust certs signed only
                                                by this CA (default
                                                "C:\\Users\\joris.vaneijden\\.docker\\ca.pem")
      --tlscert string                          Path to TLS certificate
                                                file (default
                                                "C:\\Users\\joris.vaneijden\\.docker\\cert.pem")
      --tlskey string                           Path to TLS key file
                                                (default
                                                "C:\\Users\\joris.vaneijden\\.docker\\key.pem")
      --tlsverify                               Use TLS and verify the remote
      --unregister-service                      Unregister the service
                                                and exit
  -v, --version                                 Print version information
                                                and quit

Maybe the mac build is missing it too?

@kjelle
Copy link

@kjelle kjelle commented Feb 14, 2019

Hello.
I just set --default-addr-pool 192.168.0.0/16 and my newly initiated docker swarm set the ingress network to 10.255.0.0/16.

@Jamuz
Copy link

@Jamuz Jamuz commented May 16, 2019

Hello.
I just set --default-addr-pool 192.168.0.0/16 and my newly initiated docker swarm set the ingress network to 10.255.0.0/16.

I have the same. The automatically created ingress network gets the subnet 10.255.0.0/16, but all manually created overlay networks do get the correct value set by the --default-addr-pool option. By manually re-creating the ingress network I get the correct subnet.

Automatically created ingress network gets 10.255.0.0/16:

$ docker swarm init --default-addr-pool 10.20.0.0/16
Swarm initialized: current node (t1ukoao37jotrynkucxmtrnbp) is now a manager.
... ...
$ docker network inspect ingress | grep Subnet
                    "Subnet": "10.255.0.0/16",

Manually creating a network gets 10.20.0.0/24

$ docker network create --driver overlay mynet
k5rlfr5av7glior2bm78rfqjp
$ docker inspect mynet | grep Subnet
                    "Subnet": "10.20.1.0/24",
$

I can even re-create the ingress network with correct value without using the --subnet option:

$ docker network rm ingress
WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired.
Are you sure you want to continue? [y/N] y
ingress
$ docker network create --driver overlay --ingress ingress
ojm9xnk6g55kppdvkwpc8h92u
$ docker inspect ingress | grep Subnet
                    "Subnet": "10.20.0.0/24",
$

It would be nice if the automatically created ingress network would follow the --default-addr-pool option.

@selansen
Copy link
Contributor Author

@selansen selansen commented May 16, 2019

Ingress network gets created before default IP pool init happens. Will have to take a look at on how we can get around current design. will take a look at it.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented May 16, 2019

@Jamuz perhaps you can create a ticket for that so that it doesn't get lost?

@EugenMayer
Copy link

@EugenMayer EugenMayer commented Sep 7, 2019

This merge seems to be incomplete by any means, especially docs

Beside this "help" does not help in a single way

dockerd --help | grep address-pool
      --default-address-pool pool-options       Default address pools for node specific local networks

it is even wrong, since the setting is default-address-poolS ( plural )

It is also listed here https://docs.docker.com/engine/reference/commandline/dockerd/ as singular ..
but in the example ( json ) https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file it is plural

And it does only work in plural needs to be done this way

  "default-address-pools": [ { "base": "172.30.0.0/16", "size": 24 }]

I'am running

docker --version
Docker version 19.03.2, build 6a30dfca0

Considering the existince of bip/fixed-cidr and now this and the crazyness of DNS difference between "default networks" and "docker-compose bridge based" things seem to go out of control or at least it seems to me, there is no "big plot" anymore.

Sorry for the rant. But those topics hindered me in the last 2 months and wasted way too much time considering how easy they are solved in the end - which does not remove any of it's inconsistency or hidden features / confusion after all

AlekseyMartynov added a commit to DevExpress/devextreme-ci-aws that referenced this pull request Feb 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet