Docker 18.6.1 + Swarm - ghost network

[nsteinmetz]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Hello, I have a weird behaviour since upgrading from docker 18.3.1 to 18.6.1 and traefik 1.6.4 to 1.6.6 :
• I deploy docker stacks by creating a given network per stack and connecting traefik to this stacks (via docker service update --network-add <network> <traefik>) - so far it works well
• when I deploy a new version of the stack, I remove traefik from the network (docker service update --network-rm <network> <traefik>) and destroy the stack (docker stack rm ...) and create a new one and add traefik back to the network

Actual behavior

When I redeploy the stack, I do:

if [ $($DOCKER_BIN stack ls --format "{{.Name}}" |grep ^${COMPOSE_PROJECT_NAME}$ |wc -l) -ne "0" ]; then
    $DOCKER_BIN service update --network-rm ${COMPOSE_PROJECT_NAME}_network traefik_traefik
    $DOCKER_BIN stack rm ${COMPOSE_PROJECT_NAME}
    sleep 20
    $DOCKER_BIN network prune --force
fi

$DOCKER_BIN stack deploy --compose-file docker-compose.yml --with-registry-auth ${COMPOSE_PROJECT_NAME}

if [[ ${TRAEFIK} == "docker" ]]; then
    $DOCKER_BIN service update --network-add ${COMPOSE_PROJECT_NAME}_network traefik_traefik
fi

but output is now:

...
Creating service <stack>_backoffice
failed to create service <stack>_backoffice: Error response from daemon: network <stack>_network not found

• But since the upgrade, the network is not well destroyed as traefik seems still in the network (via docker inspect network <network>)

I need to do docker stack rm traefik so that the ghost network vanish as expected and then redeploy traefik from scracth and reconnect it to other instances. Any idea on this ?

I’m trying to downgrade back to 1.6.4 to see if it’s more from docker side or traefik side... => results are the same with 1.6.4

I also noticed that for the stacks I deploy so far I did not set the overlay network as attachable - did not require it for the last 6 months - shoud I add it ? =>result is the same with attachable network.

Steps to reproduce the behavior

• Having a swarm cluster
• Having a docker-compose file with a given network
• Deploy stack
• Connect traefik to it
• Remove traefik to it
• Destroy stack
• Network is still present

sudo docker network inspect <instance_network>
[
    {
        "Name": "<instance_network>",
        "Id": "11xjt1yz38o5vtw40qbzldorx",
        "Created": "2018-08-29T16:58:57.674468644+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.49.0/24",
                    "Gateway": "10.0.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "lb-<instance_network>": {
                "Name": "<instance_network>-endpoint",
                "EndpointID": "eda2b043adb5e211003258e553a0368d5a2f306245c7609793164a8bb3e5ebe7",
                "MacAddress": "02:42:0a:00:31:04",
                "IPv4Address": "10.0.49.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4123"
        },
        "Labels": {
            "com.docker.stack.namespace": "<instance>"
        },
        "Peers": [
            {
                "Name": "517efbb11671",
                "IP": "172.16.0.5"
            }
        ]
    }
]

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 129
 Running: 123
 Paused: 0
 Stopped: 6
Images: 129
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: xqm2e3fld2vem3xali1795ug1
 Is Manager: true
 ClusterID: 7pf90t57w3oog500hniyt9rgr
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 172.16.0.5
 Manager Addresses:
  172.16.0.5:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-134-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.82GiB
Name: swarm1.*******.net
ID: SAAO:VFA7:YFS4:23ZK:TETY:LINA:ZOFO:URPG:5JYE:3SMU:YP4A:DZZV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

4 VMswith each 8vCPU / 64 Go RAM

[nsteinmetz]

By externalising the network creation on cli side and having only an external network referenced in the docker-compose file, it works again as expected. I create the network only once, attach traefik to it and period.

[nsteinmetz]

Modified my deployment script as follow:

# CHANGED - If stack exists - remove it
if [ $($DOCKER_BIN stack ls --format "{{.Name}}" |grep ^${COMPOSE_PROJECT_NAME}$ |wc -l) -ne "0" ]; then
    # $DOCKER_BIN service update --network-rm ${COMPOSE_PROJECT_NAME}_network traefik_traefik
    $DOCKER_BIN stack rm ${COMPOSE_PROJECT_NAME}
    sleep 20
    $DOCKER_BIN network prune --force
fi

# ADDED - Create network
if [ $($DOCKER_BIN network ls --format "{{.Name}}" |grep ^${COMPOSE_PROJECT_NAME}_network$ |wc -l) -eq "0" ]; then
    $DOCKER_BIN network create -d overlay --attachable ${COMPOSE_PROJECT_NAME}_network
fi

# Run stack
$DOCKER_BIN stack deploy --compose-file docker-compose.yml --with-registry-auth ${COMPOSE_PROJECT_NAME}

if [[ ${TRAEFIK} == "docker" ]]; then
    # ADDED Check if Traefik is already in the network or not
    network=`$DOCKER_BIN network ls --no-trunc |grep ${COMPOSE_PROJECT_NAME} |awk '{print $1}' |wc -l`
    if [[ $network -ne "0" ]]; then
        # Network exists, if Traefik is not already added in the network, add it - do nothing otherwise
        if [ $($DOCKER_BIN service inspect traefik_traefik --format="{{json .Spec.TaskTemplate.Networks}}" | grep `$DOCKER_BIN network ls --no-trunc |grep ${COMPOSE_PROJECT_NAME}_network |awk '{print $1}'` |wc -l) -eq "0" ]; then
            $DOCKER_BIN service update --network-add ${COMPOSE_PROJECT_NAME}_network traefik_traefik
        fi
    fi
fi

[nsteinmetz]

Close in favor of the one in swarmkit