Macvlan and bridge drivers with swarm scope are not calling remote IPAM plugin

[ishantt]

Description
The macvlan and bridge drivers with swarm scope are not calling the remote IPAM plugin.

$ sudo docker plugin ls
ID                  NAME                            DESCRIPTION                       ENABLED
169547e61097        ishant8/ipam-plugin:1.1.0-pre   Infoblox IPAM plugin for Docker   true

$ sudo docker network create macvlan-swarm --driver macvlan --scope swarm --ipam-driver ishant8/ipam-plugin:1.1.0-pre --attachable -o parent=eth2.10
eh1kyb5ln7c782cdcc7zyazua

$ sudo docker run -dit --net macvlan-swarm --name swarm alpine sh
0fca5b3022e84019fb9b024c6fd67c61a6bc44c37ace860e03c0bbe5844b363f

$ sudo docker network inspect macvlan-swarm
[
    {
        "Name": "macvlan-swarm",
        "Id": "eh1kyb5ln7c782cdcc7zyazua",
        "Created": "2017-06-23T12:38:30.516685016+03:00",
        "Scope": "swarm",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "0fca5b3022e84019fb9b024c6fd67c61a6bc44c37ace860e03c0bbe5844b363f": {
                "Name": "swarm",
                "EndpointID": "8f4cecb1a2e0c8d0bc0a518bdd8ccf17c76f01e92f01fe3e686a6148652cc550",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {},
        "Peers": [
            {
                "Name": "master-d89a069238d8",
                "IP": "40.0.0.41"
            }
        ]
    }
]

Notice the IPAM info is empty in the network inspect. There is no error throw and driver allocates IP from the default docker IPAM.

In local mode , drivers are working as expected.

$ sudo docker network create macvlan-local --driver macvlan  --ipam-driver ishant8/ipam-plugin:1.1.0-pre --attachable -o parent=eth2.20


$ sudo docker run -dit --net macvlan-local alpine sh
e45376c05ec890390ff819dd5b387f0e72b13ade2886137de851a4ca12ebcb79



$ sudo docker network inspect macvlan-local
[
    {
        "Name": "macvlan-local",
        "Id": "036a36d575c4f186beb0a0039d916d5c5f6b7f17e65faf4589490ca4ff6ae5ff",
        "Created": "2017-06-23T12:13:25.859349647+03:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "ishant8/ipam-plugin:1.1.0-pre",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.153.12.0/26",
                    "Gateway": "192.153.12.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "e45376c05ec890390ff819dd5b387f0e72b13ade2886137de851a4ca12ebcb79": {
                "Name": "eager_goldstine",
                "EndpointID": "4a1060ed11585041a6e4229304c9945717fd2c7e180e5a03de16f28aceecdfbf",
                "MacAddress": "02:42:20:1b:d7:3c",
                "IPv4Address": "192.153.12.2/26",
                "IPv6Address": ""
            }
        },
        "Options": {
            "parent": "eth2.20"
        },
        "Labels": {}
    }
]

Steps to reproduce the issue:

1. Install the remote IPAM plugin
2. Create a network with macvlan driver with swarm scope and specifying --ipam-driver as remote IPAM plugin
3. Create a container with this network
4. The IPAM calls never lands to the remote IPAM plugin and IP is allocated by the default docker IPAM

Output of docker version:

~$ sudo docker version
Client:
 Version:      17.06.0-ce-rc5
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   b7e4173
 Built:        Tue Jun 20 07:13:24 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce-rc5
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   b7e4173
 Built:        Tue Jun 20 07:12:15 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

$ sudo docker info
Containers: 14
 Running: 2
 Paused: 0
 Stopped: 12
Images: 2
Server Version: 17.06.0-ce-rc5
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 33
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: b35k153d44qox98ffjz89ig92
 Is Manager: true
 ClusterID: bvyf6n1vriwppdmmouazrn5b5
 Managers: 1
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 40.0.0.41
 Manager Addresses:
  40.0.0.41:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-22-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 5.711GiB
Name: master
ID: WZHT:JFVP:B34A:VQ6O:IKYE:PNNV:3OHO:QZ6S:RKCQ:TMXG:VQFF:ZNVA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[sanimej]

@ishantt To use macvlan in swarm mode just --scope swarm is not sufficient. macvlan needs a local node specific config, for the parent interface. So you have to create the config-only network locally and link it with the network create in the manager by using config-from. please refer to this config example here..
https://github.com/mark-church/docs/blob/master/local-scope-swarm-networking.md

[nicolaka]

@ishantt were you able to use the configs that Santhosh mentioned above?

[ishantt]

@nicolaka I was on vacation last week. I will test it today and update the issue.

[ishantt]

@sanimej using the config-only network and the attaching it with the swarm scope MACVLAN network works. The requests are landing in the IPAM plugin as expected. However the scope for the network is local which is odd for the MACVLAN driver. I understand that IPAM plugin is getting the local scope request because the network is local to the node but the MACVLAN use case is to provide global address space across the hosts. For the MACVLAN driver the local scope IPAM does not make sense because the user wants the container network on its L2 network across the hosts.

For the bridge network the request never lands in the IPAM plugin. The behaviour is as described in the issue description.

[sanimej]

@ishantt Yes, the scope of the network is local because the resource allocation is local to that node. This is ok if the user has configured the --ip-range appropriately. This also gives the flexibility to use different subnets across nodes if the underlay can route the subnets correctly.

In cases where user doesn't want to configure --ip-range one option would be to use the --ipam-opt to pass a hint to the IPAM driver.

[ishantt]

Thanks @sanimej . I will implement MACVLAN in swarm mode based on this design.

[thaJeztah]

Following the conversation it looks like the issue is resolved, so i'll close, but feel free to continue the conversation.

@sanimej do we need more documentation on this subject?

[sanimej]

@thaJeztah I think its worth mentioning in the documentation. Do you know if we have any documentation yet for swarm mode macvlan ? I couldn't find any from a quick search.

[thaJeztah]

@sanimej hm.. good point, we may not have that yet; can you open an issue in the docs issue tracker? https://github.com/docker/docker.github.io/issues