docker-compose fails when trying to map many ports

[leeflix]

I am trying to map as many ports as possible (1001-65535) of my container to the host machine, but docker-compose build fails in the last steps.

This is my docker-compose.yaml that works:

version: '3.8'

services:
  frapi:
    restart: always
    build: .
    ports:
      - "8080:8080/tcp"
      - "50000-50009:50000-50009/udp"

When I try to expose the ports 1001-65535 instead of just 50000-50009 it fails:

version: '3.8'

services:
  frapi:
    restart: always
    build: .
    ports:
      - "1001-8079:1001-8079/udp"
      - "8080:8080/tcp"
      - "8081-65535:8081-65535/udp"

Describe the results you received:

root@...# docker-compose build --no-cache && docker-compose up -d --force-recreate
...
Step 10/10 : CMD ["-m", "openapi_server"]
 ---> Running in 9021e8665944
Removing intermediate container 9021e8665944
 ---> d778b1f5e103
Successfully built d778b1f5e103
Successfully tagged frapi_flask_frapi:latest
WARNING: Found orphan containers (frapi_flask_service_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating frapi_flask_frapi_1 ...
Recreating frapi_flask_frapi_1 ... error

ERROR: for frapi_flask_frapi_1  Cannot start service frapi: driver failed programming external connectivity on endpoint frapi_flask_frapi_1 (6f152bdc1598106500fc0be22db5c7281297eb3921468424df10741d59eae472): Error starting userland proxy:

ERROR: for frapi  Cannot start service frapi: driver failed programming external connectivity on endpoint frapi_flask_frapi_1 (6f152bdc1598106500fc0be22db5c7281297eb3921468424df10741d59eae472): Error starting userland proxy:
ERROR: Encountered errors while bringing up the project
root@...

Describe the results you expected:

I expected that the container is up and running, but it is not.

Output of docker compose version:

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 19
  Running: 0
  Paused: 0
  Stopped: 19
 Images: 777
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.15.0
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 8GiB
 Name: h2899006.stratoserver.net
 ID: AIE6:LG6K:WIBV:GCCE:3EAT:KR72:JIX6:JC7J:MEBM:HNHQ:TAY2:4DBM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[snooops]

i can confirm. I just wanted to open the same issue.

Description: Ubuntu 20.04.3 LTS
docker-compose version 1.26.0, build d445165

in my case i wanted to open 8000-8500/udp. it fails with the same error message. I checked if there are any ports open with netstat -tulpen and double checked if something magical else is doing stuff with:

for i in {8000..8500}; do echo -e $i; timeout 0.1 netcat -u -l -p $i; done

all ports could be created

[Msprg]

Same here.

Docker on RPi 4 on RPi-OS x32,

$ docker -v
Docker version 20.10.17, build 100c701
$ docker-compose -v
docker-compose version 1.21.0, build unknown
$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

Server:
 Containers: 11
  Running: 2
  Paused: 0
  Stopped: 9
 Images: 98
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.103-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 7.713GiB
 Name: RPi-MultiServer
 ID: FAQN:I3RF:3C2A:IXAL:6HPM:2LEX:T45E:K63F:MITJ:RUM2:BOSC:VRZ3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

When I docker-compose up a YAML with more than a few ports (~100+), it times out.

However after the timeout, I can see dockerd doing something, taking 20~30% of CPU. Given that I wait long enough, supposedly depending on the number of ports, possibly even hours, it WILL finish its job and I can issue other compose commands. Or alternatively, I have to restart dockerd to "abort" the process if I need to do something with compose command right away.

Now the only solutions (more like workarounds) I can see working so far, are:

1. Extend the timeout value, so that docker-compose will wait for dockerd to finish the job properly.
2. Lower number of ports.

Now I don't know why isn't dockerd using more CPU to - presumably - finish quicker, but I'm not sure how to find the bottleneck.

Hope this sheds at least a tiny bit of light on what's happening behind the scenes.

[ndeloof]

Can you please confirm you can run a comparable container with a bunch of ports exposed on host using plain docker run -p RANGE:RANGE ... and that this issue is a Compose one, not a limitation for docker engine in some circumstances ?

[nkartem]

If I use the driver host do not have a problem with the starting container and port range.

cat docker-compose.yaml

version: '3.1'

services:

  asterisk:
    image: nkartem/asterisk
    container_name: asterisk
    restart: always
    command: systemctl start asterisk
    network_mode: "host"

[ndeloof]

If you get this issue, please try running your service image with docker run -p RANGE:RANGE ... (doesn't matter it can't run isolated, just an attempt to understand if this is a docker engine limitation or compose bug)

[nkartem]

sudo docker run -it -d -p 10000-20000:10000-20000/udp -p 5022:5022 nkartem/asterisk

It`s not working too. Docker used all RAM and the host stopped responding.

[ndeloof]

@thaJeztah are you aware of a known issue with moby on this topic?

[thaJeztah]

Yes, this is a known issue; it’s slightly better on Docker Desktop as we disable the userland-proxy on Desktop (which would run 1 proxy per port 😬), but libnetwork creates an iptables rule for every port mapped, not a port-range.
If you're running on Linux, you can opt to use host networking for specific cases (VOIP etc), but make sure you do so with trusted workloads only, as doing so disables the networking sandbox of the container (the container is using the host's network stack, so won't have its own networking namespace, or its own IP-address). Host networking is not currently supported on Docker Desktop (you can use it, but it would use the VM's host network, not the macOS/Windows machine's host). Disabling the userland-proxy in the daemon configuration also helps (but likely will still be an issue with a large port-range)

Related issues in moby (there’s probably more) that may contain useful information;

• user land proxy uses all RAM memory when exposing a big range of ports moby/moby#11185
• Spawning 12,000 docker-proxy instances isn't sensible nor expected moby/moby#16284
• when exposing a port range, individual iptables rules instead of port range  moby/moby#19017

I think there was a PR once to update libnetwork to use a range of ports for port-mapping (instead of individual ports), but that never made it in (contributions welcome for that though, but may not be an easy job).

[ndeloof]

Closing as "not a compose issue". Please follow relevant Moby issues listed