Ambiguous error message when creating a container with more than one network

[rgov]

Description

The documentation is unclear on whether multiple networks can be specified when executing docker run.

There are old issues like this one that claim a container can only be started with one network, or this StackOverflow question whose answer links to a no-longer-existent page on docker.com that says "Docker only allows a single network to be specified with the docker run command."

Is this still the case? The documentation is silent on the topic, and the error messages leave it ambiguous.

Reproduce

$ docker run --rm -it --network foo --network bar alpine:latest
docker: Error response from daemon: Container cannot be connected to network endpoints: foo, bar

Expected behavior

The error message and documentation should be clear about whether multiple networks are supported.

If multiple networks are not supported, and I pass multiple --network flags, please display a concise error message like "Only one network can be specified when starting a container. Additional networks can be attached later with docker network connect."

If multiple networks are supported, this must have been a change in the last few years. Some sentence about this in the documentation confirming which version this was supported in would be useful.

docker version

Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:00:48 2022
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:09 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.12.2)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.253-tegra
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 31.18GiB
 Name: ifcb110
 ID: ZO22:42WT:U6GF:LMQP:JE2S:ZZEA:I2FU:XTJL:BZCB:AU2C:GCS2:Q6CZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[corhere]

The documentation is quite explicit, actually. You make a good point about the quality of the error message, though.

@thaJeztah do you remember why the decision was made to disable the ability to create a container with more than one network attached? #19560 only says "to avoid confusion it was recommended to disable it."

[rgov]

The documentation is quite explicit, actually.

Thanks for pointing that out. It seems like it could be moved earlier. I would not expect to find this information under the heading "IP address and hostname". What about in the documentation for docker run --network?

(It is very reasonable to assume that, absent information to the contrary in the documentation for --network, given that a container can be connected to multiple networks, that the option can be repeated, like --volume.)

Also for what it's worth if you search the docs with the built-in search widget for the phrase "multiple networks" the page you linked is not within the first page of results, so it's a little hard to find through search.

[thaJeztah]

Thanks for the ping; yes the error message could use some tweaking, but (story-telling time!):

TL;DR

very likely multiple networks can be supported on docker run (and docker create), but it needs to be verified (are there possible race-conditions in IP assignment etc etc), which would allow the error-message to go away entirely.

The long answer as to "why multiple networks on container create are currently not supported";

Originally, there was only 1 network (the default "bridge" network), but the --net ("network mode") option allowed the container to use a different mode (host or none). At some point, network options, such as --mac-address were added to docker run / docker create: #8390

When custom networks were added, the existing --net (later --network) option was used to specify the custom network to attach to; management commands were added to allow the container to be attached to additional networks using docker network connect.

Now that custom networks were possible; the (obvious) question was "why not allow attaching to multiple networks on docker run?" #17750. Unfortunately, network-related options (like --mac-address) somewhat painted us in a corner; #17796 (comment). I recall discussing with the libnetwork team to see if UX was the only limitation (ISTR there were some other factors), but I probably didn't push hard enough at the time (I did find some issues in this area; #19418, which actually led to the introduction of this error message in #19560).

More options were added, such as --network-alias (to reimplement the old --link option), and while "some" discussion happened on "multiple networks" (#19242 (comment)), things were kept as-is (only a single network on "create", use docker network connect for multiple networks).

Fast-forward some time; more options were needed for networks, and a PR was opened to add a --network-opt flag (#27638). By then, I was more ingrained into the project, dug into "is it really not possible to have multiple networks?", and decided to ultimately decided to put my foot down (after I raised similar discussions for volumes (#28527)). #27638 (comment)

Thinking about this; should --network-opt be separate from --network? We've made that mistake with --volume-driver, and because of that, only a single driver can be specified on docker run. The advanced (--mount) syntax that's present on docker service create resolves that, but is not available yet for docker run.

Would it make sense to have an advanced syntax for --network that allows specifying these options per-network? This would also allow specifying multiple networks on docker run (which we decided not to do, because per-network options may be needed), e.g.;

--network name=my-net,opt=foo=bar,opt=foo2=baz

I haven't checked in-depth what kind of options are passed here, but thought to mention this, before it's too late 😄

So, I wrote up a proposal for an alternative UX; #31964 - while this wouldn't fix the problem, it at least would not paint ourselves further into a corner. This syntax was implemented for Swarm services (docker/cli#62). The equivalent for docker create / docker run stalled, but I carried it at some point in docker/cli#1767

On that PR, the "single network" restriction was removed from the CLI, delegating the validation to the daemon (API); this was from my estimation that the API should be able to support multiple networks, but further investigating was needed to confirm that, and as it was getting close to a release, I kept the "single network only" validation in place on the daemon side docker/cli#1767 (comment)

To support multiple networks; this part will have to be removed from the engine, but otherwise (I think) that all works;

moby/daemon/create.go

Lines 308 to 314 in c7105e3

	if len(nwConfig.EndpointsConfig) > 1 {
	l := make([]string, 0, len(nwConfig.EndpointsConfig))
	for k := range nwConfig.EndpointsConfig {
	l = append(l, k)
	}
	return errors.Errorf("Container cannot be connected to network endpoints: %s", strings.Join(l, ", "))
	}

This should definitely be looked into, as it would also simplify how docker compose creates containers / services; currently it has to create the containers, then "loop" through all of them to set the correct network aliases for networks they're attached to (which could likely all be done immediately when creating the container).

[akerouanton]

• api: Allow ContainerCreate to take several EndpointsConfig for >= API 1.44 #45906 was merge last week. The error message has been revised in that PR, so let me close this ticket.