Error response from daemon: network docker_gwbridge has active endpoints

[michaeldayCA]

Description

Attempting to follow the steps at https://success.docker.com/article/how-do-i-change-the-docker-gwbridge-address to remove the docker_gwbridge network fails with message: "Error response from daemon: network docker_gwbridge id xxxxx has active endpoints"

Steps to reproduce the issue:

1. Stop all running containers
2. docker network rm docker_gwbridge

Describe the results you received:

"Error response from daemon: network docker_gwbridge id xxxxx has active endpoints"

Describe the results you expected:

docker_gwbridge is successfully removed

Additional information you deem important (e.g. issue happens only occasionally):

Stopping and starting the docker daemon did not help. To succeed, I had to use

docker network disconnect --force docker_gwbridge gateway_ingress-sbox

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 22
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: skgf4dv4ux9h1hx3e72mdbfd8
 Is Manager: true
 ClusterID: zn70moohy7c4flpt96g4begj7
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.7.53.227
 Manager Addresses:
  10.7.53.227:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26GiB
Name: docker_host
ID: DRLU:OX6M:MHCB:46AZ:ZAHV:XLCW:272G:PUZ4:P62L:UILM:RIVX:6JIR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

VM running on VMWare ESXI/VCenter 6

Complete steps for troubleshooting:

[centos@docker_host ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[centos@docker_host ~]$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[centos@docker_host ~]$ sudo docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[centos@docker_host ~]$ docker network rm docker_gwbridge
Error response from daemon: network docker_gwbridge id 804653507da34e773059ff28f5dfb1055cfbd7c16e01808b372c810af2e6a58c has active endpoints
[centos@docker_host ~]$ docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "804653507da34e773059ff28f5dfb1055cfbd7c16e01808b372c810af2e6a58c",
        "Created": "2018-05-01T15:28:49.63617387-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "dab9a444acf208de35b164054db597d4c658117a5dba90536c65c6f164fb589d",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]
[centos@docker_host ~]$ docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
Deleted Containers:
05b07339348a2fc1dd6bac9fbd59e5940b39171d5f96d8ce075bba796e7fdb1e
...
7c06b626377174003790cd32f7f217dd7add6b7fba92c121d3922057addf379a

Deleted Images:
untagged: dockersamples/visualizer@sha256:bc680132f772cb44062795c514570db2f0b6f91063bc3afa2386edaaa0ef0b20
...
deleted: sha256:5bef08742407efd622d243692b79ba0055383bbce12900324f75e56f589aedb0

Total reclaimed space: 450MB
[centos@docker_host ~]$ docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "804653507da34e773059ff28f5dfb1055cfbd7c16e01808b372c810af2e6a58c",
        "Created": "2018-05-01T15:28:49.63617387-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "dab9a444acf208de35b164054db597d4c658117a5dba90536c65c6f164fb589d",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]
[centos@docker_host ~]$ docker network rm docker_gwbridge
Error response from daemon: network docker_gwbridge id 804653507da34e773059ff28f5dfb1055cfbd7c16e01808b372c810af2e6a58c has active endpoints
[centos@docker_host ~]$ docker stop ingress-sbox
Error response from daemon: No such container: ingress-sbox
[centos@docker_host ~]$ docker stop gateway_ingress-sbox
Error response from daemon: No such container: gateway_ingress-sbox
[centos@docker_host ~]$ docker network disconnect --force docker_gwbridge ingress-sbox
Error response from daemon: endpoint ingress-sbox not found
[centos@docker_host ~]$ docker network disconnect --force docker_gwbridge gateway_ingress-sbox
[centos@docker_host ~]$ docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "804653507da34e773059ff28f5dfb1055cfbd7c16e01808b372c810af2e6a58c",
        "Created": "2018-05-01T15:28:49.63617387-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]
[centos@docker_host ~]$ docker network rm docker_gwbridge
docker_gwbridge
[centos@docker_host ~]$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
d9432128aa1c        bridge              bridge              local
acc20ffb4d59        host                host                local
xrad9row46w7        ingress             overlay             swarm
bed9358defe1        none                null                local

[thaJeztah]

ping @ctelfer

[ctelfer]

Hey, @michaeldayCA . Unfortunately, this article is a bit out of date. The update is actually under review, but not pushed yet. Changing the IP address of docker_gwbridge with current versions of docker is similar, but not the same. It requires doing the following for each node in the swarm (for which one wants to change the IP address):

1. stop any classic containers in docker_gwbridge (i.e., those not started by swarm)
2. make the node leave the swarm (this will rebalance service tasks to other nodes)
3. remove the docker_gwbridge network
4. recreate the docker_gwbridge network using the desired IP address
5. have the node rejoin the swarm
6. restart the classic containers that were stopped

For 1. above in order to get the list of classic containers you can do:

gwbridge_containers=$(docker network inspect docker_gwbridge --format '{{range $k, $v := .Containers}}{{$k}}{{printf "\n"}}{{end}}' |xargs -I{} docker container ls -f is-task=false -f id={} --format {{.Names}})

This is similar to the current documentation but filters out any task containers. Removing and recreating the docker_gwbridge network is the same as before as is restarting the classic containers. Joining and leaving the swarm network is as in https://docs.docker.com/engine/swarm/join-nodes/#join-as-a-worker-node and https://docs.docker.com/engine/reference/commandline/swarm_leave/.

[thaJeztah]

@ctelfer could you open a ticket (or PR) in the documentation issue tracker?

[thaJeztah]

Looks like it was documented on success.docker.com; https://success.docker.com/article/how-do-i-change-the-docker-gwbridge-address, but perhaps we should also add it do the docs itself