Containers attached to custom network don't start

[xwillq]

Description

I have network where I define custom subnet, ip-range and gateway. After updating to Docker Desktop 4.20 (from 4.19, though I don't remember exact version), starting containers attached to this network gives error docker: Error response from daemon: invalid bit range [0, 16777215).. Before update everything was working fine.

Reproduce

All of the steps were performed after reseting to factory defaults, without touching anything else.

1. docker network create --subnet 172.92.0.0/16 --ip-range 172.92.0.0/8 --gateway 172.92.0.1 main
2. docker run --network main alpine

Second command gives docker: Error response from daemon: invalid bit range [0, 16777215).

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.20.1 (110738)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:50:59 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/xwillq/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/xwillq/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/xwillq/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/xwillq/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/xwillq/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/xwillq/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/xwillq/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/xwillq/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: ac0be214-9ba2-41ad-9797-00a01bec94d7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

366DA27F-A3A9-4895-92C6-1B9F3B20F92C/20230609151052

Additional Info

No response

[akerouanton]

Hi @xwillq, I see two issues with your docker network create command:

1. You're specifying an --ip-range which is bigger (ie. /8) than its subnet (ie. /16).
2. The --ip-range prefix have some of its host bits set. Said another way: it should be 172.0.0.0/8 (which would fail anyway, because it overlaps with the default address pools).

Docker Desktop v4.20 ships a new version of the Engine (ie. v24). This new Engine version contains some changes related to the way we do IPAM. Although we took great care to not break anything, it looks like we didn't anticipate this specific edge-case where IPAM config is wrong.

I think we won't revert back to the old behavior. However, I admit this user experience isn't great: your docker network create should fail with an error message explaining why, instead of letting you carry on. I'll open a PR for that.

In previous Desktop/Engine versions, your network was producing IP addresses in the range 172.92.0.1/16 -> 172.92.255.254/16. Let alone the off-by-two error, you should be able to create an equivalent network with docker network create --subnet=172.92.0.0/16 main. Let me know if that doesn't work for you.

As your network create command is equivalent to the one I suggest above, I'm a bit curious why you needed this --ip-range parameter in the first place. It'd be helpful if you can share some details.

[xwillq]

Hi, thanks for explanation! I don't understand second issue you outlined. If I set --ip-range to 172.0.0.0/8, won't it break because of first issue, even if it wouldn't overlap with default ips?

I think we won't revert back to the old behavior. However, I admit this user experience isn't great: your docker network create should fail with an error message explaining why, instead of letting you carry on. I'll open a PR for that.

Yeah, makes sense. Preserving undefined behavior caused by broken configs should't be a priority. And if docker network create validated subnets and ip ranges it would be a nice safety net to prevent issues like that.

As your network create command is equivalent to the one I suggest above, I'm a bit curious why you needed this --ip-range parameter in the first place. It'd be helpful if you can share some details.

As far as I understand, docker network create --subnet 172.92.0.0/16 --ip-range 172.92.0.0/24 will create a network that assigns ips in in 172.92.0.xxx range, but allows ips in 172.92.xxx.xxx range.
I needed a way to always resolve *.test domains to one particular container, and I couldn't find a way to do it with docker networking tools. To solve this issue, I assigned static ip to one of my containers and embedded dnsmasq into container that has to communicate with the other one.
I wanted to specify different subnets in --subnet and --ip-range to ensure that that static ip won't be taken by another container.

If you want to see more concrete example, here is my repo. You can find command to create a network in Makefile in a root directory, and how I use it in reverse-proxy/compose.yml and reverse-proxy/images.

[akerouanton]

@xwillq Sorry to not come back to you earlier. The Engine PR is moving and should now be in its latest review cycle. It'll add validation steps to prevent users to create networks with invalid config. Hopefully it'll be released in Engine v25.0 and available in Docker Desktop in one or two releases.

I don't understand second issue you outlined. If I set --ip-range to 172.0.0.0/8, won't it break because of first issue, even if it wouldn't overlap with default ips?

Yes, absolutely! I totally ignored the first issue when I said "it should be 172.0.0.0/8. The important part is that, for a given block size, only the higher bits matching the block size should be set, other bits need to be 0. For instance let's say you choose an /8, then 172.0.0.0/8 is valid because 172. matches the first 8 bits. 172.92.0.0/8 is invalid because .92. comes after the 8 first bit.

As far as I understand, docker network create --subnet 172.92.0.0/16 --ip-range 172.92.0.0/24 will create a network that assigns ips in in 172.92.0.xxx range, but allows ips in 172.92.xxx.xxx range. [...] I wanted to specify different subnets in --subnet and --ip-range to ensure that that static ip won't be taken by another container.

That makes sense!

I now realize there's potentially a third issue with your initial docker network create command: the subnet 172.92.0.0/16 isn't part of the Private Address Space defined by RFC1918. So, you might see network connectivity issues if you ever need to communicate with a server in this address block 😉

[thaJeztah]

• [24.0 backport] ipam: Replace ChildSubnet with parent Subnet when its mask is bigger moby/moby#46482 was merged, and will be included in the v24.0.7 release of docker engine