Skip redundant endpoints in "network inspect"

[mavenugo]

With moby/moby#21160, docker engine will return all the endpoints (local, remote & stale endpoints). The remote and stale endpoints will have be represented as ep-{endpoint-id} . Example output from a host that doesn't own this endpoint (which represents the remote container).

$ sudo docker network inspect test2
[
    {
        "Name": "test2",
        "Id": "eaa6a4786dcba692bf240f4199848d565b2948c0481d1987051d5ce4eef080b6",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1/24"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "ep-84dcacdbed8ed7fcd2004d43612bb5bba91aceaf12c77ba6943e2c075e9d22dd": {
                "Name": "ep3",
                "EndpointID": "84dcacdbed8ed7fcd2004d43612bb5bba91aceaf12c77ba6943e2c075e9d22dd",
                "MacAddress": "02:42:0a:00:01:02",
                "IPv4Address": "10.0.1.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

This would cause duplicate endpoints in the docker network inspect output when executed in a swarm environment.

The fix is to skip such duplicate endpoints. But make sure the stale endpoints (which will not have a duplicate endpoint-id) should be returned.

[dongluochen]

@mavenugo Currently Swarm each engine overlay network manifest includes every container local, remote, or stale. Say there are N nodes with M containers in an overlay network, docker inspect network would have to filter N * M entries to get unique results. It seems a waste on resource. Is it ok that swarm doesn't restore remote/stale? I don't see them useful for Swarm. Here is an example output.

ubuntu@ip-172-19-109-234:~$ docker -H manager:3375 network inspect hello-net
[
    {
        "Name": "hello-net",
        "Id": "13af04cf34937555d6e3622a3abdc9bacfe53d80f2f5174af5338671fd56f18b",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1/24"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "548c8eadb297ca11f787848f0aa44c24891d72b575f2264e934ecf92e485ce12": {
                "Name": "bb3",
                "EndpointID": "0dabe817dae8e69d6e6285ae453bc7b81b2f4148e5e9ffd040704e1eef5ba6af",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            },
            "cf52ec56327c4143df6791998b9a8e2ec5d5798928c74927dae752bbe5b0ee7a": {
                "Name": "bb1",
                "EndpointID": "2d3f5a3cc6983886471ec5dab3e71978441d6eefca73b5feaaa509bdcc47736d",
                "MacAddress": "02:42:0a:00:01:02",
                "IPv4Address": "10.0.1.2/24",
                "IPv6Address": ""
            },
            "ep-0dabe817dae8e69d6e6285ae453bc7b81b2f4148e5e9ffd040704e1eef5ba6af": {
                "Name": "bb3",
                "EndpointID": "0dabe817dae8e69d6e6285ae453bc7b81b2f4148e5e9ffd040704e1eef5ba6af",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            },
            "ep-2d3f5a3cc6983886471ec5dab3e71978441d6eefca73b5feaaa509bdcc47736d": {
                "Name": "bb1",
                "EndpointID": "2d3f5a3cc6983886471ec5dab3e71978441d6eefca73b5feaaa509bdcc47736d",
                "MacAddress": "02:42:0a:00:01:02",
                "IPv4Address": "10.0.1.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

[dongluochen]

@mavenugo I feel like listing remote and stale containers should be optional. Doesn't it make sense to have default network command not to return them? You can add an option to show them. It'd make it clean for existing tools.

[mavenugo]

@dongluochen Swarm master infact reflects the cluster state and not the local state. But the Stale endpoints are never seen either in swarm or engine level... The main intention of moby/moby#21160 is to identify a stale endpoint. unfortunately at the engine level, it will not know if an endpoint is a remote endpoint or a stale endpoint. But Swarm can (due to his global view of containers that are alive). And that is the exact purpose of this Issue.

Swarm has the ability to see all the containers that are connected to an overlay network across the cluster and hence it can filter out any redundant ep-{endpoint} that has a corresponding active container. Any stale endpoint with a ep-{endpoint} without a corresponding active container must be shown in the docker network inspect output so that it can be manually removed by the user using docker network disconnect -f.

That is the reason this is not an optional entity.

[dongluochen]

But Swarm can (due to his global view of containers that are alive).

@mavenugo I'm not sure of that. Today overlay network is control with cluster-store option. An overlay network can be bigger than a Swarm cluster. Nodes in an overlay network know each other, while Swarm doesn't.

The weird thing here is that a node is reporting endpoints that doesn't belong to it. Also the search space jumps from linear to quadratic.

[mavenugo]

I'm not sure of that. Today overlay network is control with cluster-store option. An overlay network can be bigger than a Swarm cluster. Nodes in an overlay network know each other, while Swarm doesn't.

@dongluochen that's absolutely fine. If someone is deploying that way, then swarm's docker network inspect will also contain the ep-{endpoint} from the node that is not controlled by swarm. The idea of this feature is to make sure we provide all the tools for the user to visualize what is seen by the network subsystem.

The weird thing here is that a node is reporting endpoints that doesn't belong to it. Also the search space jumps from linear to quadratic.

That's not true, in a multi-host scenario, every endpoint that is attached to a network is in global space due to the necessary service discovery and endpoints doesn't belong to a particular node. Whereas a container certainly belong to a particular node at a given point in time. Since an endpoint is 1:1 mapped to container, it inherits the container properties and the states are very similar for a particular network. But once we move to a service level construct, service becomes a global entity and this discussion doesn't arise.

[bklau]

@mavenugo : Currently, I 'm using Docker 1.12.2 with standalone Swarm. I'm not using Swarm mode.
I created a overlay network : "docker network create -d "overlay" ....." and then attach the overlay to several running container like so "docker connect..." on different docker hosts. When I do a "docker inspect, I can see all the containers listed without "ep-" prefix:
"Containers": {
"5413758c925a5554d1f7448f9ddab725a853835a00b203029abc1ab4f561076f": {
"Name": "cocky_roentgen",
"EndpointID": "10b4a9f2fcdeef5c9df992ab058b092318578eefedab590f1effb539f92b3a5d",
"MacAddress": "02:42:0a:00:01:03",
"IPv4Address": "10.0.1.3/24",
"IPv6Address": ""
},
...
}

The strange thing is that when I "disconnect" anyone of them from the overlay, and I run "docker inspect...", again immediately, I see something like this:
Example: I disconnect "5413758c925a5554d1f7448f9ddab725a853835a00b203029abc1ab4f561076f" from the overlay example above and in its place when I do a "network inspect" again, I got:
"Containers": {
"ep-10b4a9f2fcdeef5c9df992ab058b092318578eefedab590f1effb539f92b3a5d": {
"Name": "cocky_roentgen",
"EndpointID": "10b4a9f2fcdeef5c9df992ab058b092318578eefedab590f1effb539f92b3a5d",
"MacAddress": "02:42:0a:00:01:03",
"IPv4Address": "10.0.1.3/24",
"IPv6Address": ""
},
...
}

If run "docket network inspect" again 1-2 secs later, the "ep-10b4a9f2fcdeef5c9df992ab058b092318578eefedab590f1effb539f92b3a5d" entry went away...

Q: Is this the correct behaviour?. Has any of the subsequent release of 1.12.x or even 1.13 fixed this?

Thx,

[nishanttotla]

@bklau I will fix that issue in #2690

The problem is that the Swarm refresh loop uses network ls which has stopped returning container information.