[BUG] V2 Compose can't attach multiple networks reliably

[nel0x]

Description

When multiple networks (~ more than 4) are defined in docker-compose.ymland attached to a service with docker compose up, often will one network - seemingly randomly and changing with each container recreation - not be attached to the container (docker inspect container).

If you recreate with docker compose up --force-recreate, it's very likely that missing, unattached network will now be another one; only rarely will all networks be attached as expected. However, even with 20 networks, it's always only one network that doesn't get attached, the rest are fine.

This problem only occurs with V2 docker compose ! Starting your container/stack with the deprecated V1 docker-compose works perfectly fine.

Steps To Reproduce

1. Create networks

docker network create -d bridge net1
docker network create -d bridge net2
docker network create -d bridge net3
docker network create -d bridge net4
docker network create -d bridge net5

2. Populate docker-compose.yml and start container

mkdir /tmp/hello-world & cd /tmp/hello-world
vi docker-compose.yml && docker compose up

version: '3'

networks:
  net1:
    external: true
  net2:
    external: true
  net3:
    external: true
  net4:
    external: true
  net5:
    external: true

services:
  hello-world:
    image: hello-world
    container_name: hello-world
    networks:
      - net1
      - net2
      - net3
      - net4
      - net5

Compose Version

# docker compose version
Docker Compose version v2.19.1

# docker-compose version
docker-py version: 5.0.3
CPython version: 3.11.3
OpenSSL version: OpenSSL 3.0.9 30 May 2023

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 1
 Server Version: 24.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.3.6-200.fc38.x86_64
 Operating System: Fedora Linux 38 (Thirty Eight)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 3.716GiB
 Name: fedora-4gb-hel1-1
 ID: ff99be82-ffd8-4505-84bb-9d4638ca5779
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

Tested this with Fedora 38 as well as Debian 11, also with different Docker images; Behavior remains the same.

[bruceesmith]

I've found this problem more likely when multiple compose files (-f arguments) are on the docker compose command line.

Create 3 separate Docker Compose specifications - each defines its own user-defined bridge network, the third also connects to the bridge network of each of the first two. In detail

fred.yml

services:
fred:
    hostname: fred
    image: bash:latest
    command: [ "bin/sleep","1h" ]
    networks:
    - fred
networks:
fred:

harry.yml

services:
harry:
    hostname: harry
    image: bash:latest
    command: [ "bin/sleep","1h" ]
    networks:
    - harry
networks:
harry:

bill.yml

services:
bill:
    hostname: bill
    image: bash:latest
    command: [ "bin/sleep","1h" ]
    networks:
    - fred
    - bill
    - harry
networks:
fred:
bill:
harry:

Bring up the networks and the containers

docker compose -f bill.yml -f fred.yml -f harry.yml -p x up -d

It seems that the behaviour varies depending on the order of the -f arguments, and the behaviour is non-deterministic (so, sometimes all networks are established as expected, sometimes they are not)

To reveal the incomplete network setup, do two commands

docker exec x-bill-1 ping harry
docker exec x-bill-1 ping fred

• If network setup is complete, both will succeed
• If network setup is incomplete, one or the other ping commands will fail (e.g. bad address 'fred')
• When one ping fails, docker network inspect will show that the failing network is not connected to the container x-bill-1

[Schmoho]

+1
I have been noticing this too lately

[alfonx]

+1 We suffer from this bug, too

[ndeloof]

tested with latest codebase, can't reproduce.
I guess this has been fixed by #10778
Can you please upgrade to v2.20 and confirm this issue is fixed ?

[nel0x]

@ndeloof your right! It has been fixed in v2.20 by #10778
No issues with multiple networks anymore. Thank you!
Should soon be available in the stable repo for everybody

[WhoAmI0501]

Good Morning,

Should soon be available in the stable repo for everybody

May you can get a bit more detailed with this statement? The Release v2.20 is out since 5 days and it is still not in the Debian repositories of Docker. Neither test nor stable.

With respect to all of the docker compose developers - this is a really cool tool, but this bug fucked up so many infrastructures in the last days and i am currently struggling with version pinning to v2.18.1 and so on. This is really not sustainable and very annoying for me as a sysadmin.

May you can take a look whats going on there and why the release isn't published in the apt repositories? Thank you!

[ndeloof]

@WhoAmI0501 the debian package are generated as part of docker engine release and won't get updated by a docker compose release. In the meantime you can install by downloading binaries from https://github.com/docker/compose/releases/tag/v2.20.0

[bruceesmith]

The speed at which this defect was resolved is impressive. Downloading binaries containing the fix is fine for test systems. However, given the non-deterministic behaviour of Compose with this defect present, in a global context it becomes a game of whack-a-mole until such time as the fix emerges from the various package repositories. I humbly suggest that an updated Docker Engine release that contains this fix should be given serious consideration.

[mdoerfel]

#10778 mentions "ServiceConfig::NetworksByPriority is neither deterministic nor stable when networks have the same priority."

A workaround for us was to explicitly assign disjunct priorities to the networks of a service. Adapted configuration from the test-case above:

[...]
services:
  hello-world:
    image: hello-world
    container_name: hello-world
    networks:
      net1:
        priority: 10
      net2:
        priority: 20
      net3:
        priority: 30
      net4:
        priority: 40
      net5:
        priority: 50

[nel0x]

PS: v2 is definitely released for Debian/Ubuntu now