dockerd fails to start with --iptables=false --ip6tables

[rhansen]

Description

When started with --iptables=false and --ip6tables, dockerd fails with:

failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: cannot create new chains, EnableIPTable is disabled

Reproduce

dockerd --ipv6 --fixed-cidr-v6=fd00::/64 --iptables=false --experimental --ip6tables

Expected behavior

dockerd should start, and create IPv6 rules but not IPv4 rules.

docker version

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:44 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 1159
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-26-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 23.46GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/16, Size: 24
   Base: 172.18.0.0/15, Size: 23
   Base: 172.20.0.0/14, Size: 22
   Base: 172.24.0.0/13, Size: 21
   Base: fd01::/64, Size: 64

Additional Info

The following change should fix it (I'll open a pull request after I write a regression test):

diff --git a/libnetwork/drivers/bridge/setup_ip_tables_linux.go b/libnetwork/drivers/bridge/setup_ip_tables_linux.go
index c98781f3b0..2d3ed57d9d 100644
--- a/libnetwork/drivers/bridge/setup_ip_tables_linux.go
+++ b/libnetwork/drivers/bridge/setup_ip_tables_linux.go
@@ -31,11 +31,6 @@ const (
 )
 
 func setupIPChains(config configuration, version iptables.IPVersion) (natChain *iptables.ChainInfo, filterChain *iptables.ChainInfo, isolationChain1 *iptables.ChainInfo, isolationChain2 *iptables.ChainInfo, retErr error) {
-	// Sanity check.
-	if !config.EnableIPTables {
-		return nil, nil, nil, nil, errors.New("cannot create new chains, EnableIPTable is disabled")
-	}
-
 	hairpinMode := !config.EnableUserlandProxy
 
  	iptable := iptables.GetIptable(version)

[akerouanton]

Could you elaborate on why you want iptables disabled but ip6tables enabled?

[rhansen]

I want to manually control IPv4 rules but I'm OK with Docker controlling the IPv6 rules.

[akerouanton]

I want to manually control IPv4 rules but I'm OK with Docker controlling the IPv6 rules.

Can you elaborate on why? Is it due to some issues you face with the iptables rules Docker creates?

[thaJeztah]

Oh! Looks like I missed that a discussion was happening here as well; I left some comments related to this in #46468 (comment)

[rhansen]

I'm mostly experimenting right now. I'm working on adding some outgoing NAT rule unit tests in preparation for fixing feature request #46469, and noticed this issue. I didn't expect it to be so controversial.

One reason to disable iptables (IPv4) but enable ip6tables (IPv6) is running multiple Docker daemons, where only one of the daemons does IPv6.

[rhansen]

(@thaJeztah I'm moving the conversation in #46468 (comment) here because deciding the semantics of the --iptables flag deserves greater visibility, and in case it is decided that that PR is not the right way forward)

we need to be careful here

Agreed.

Regardless of the decision reached, I think the documentation for the --iptables flag should be updated to clarify its intended purpose.

IF we would working from a greenfield state

In a way we are in a greenfield state right now. Passing --iptables=false --ip6tables=true is invalid due to this check, so nobody uses it. We can apply any semantics we want to that particular combination, as long as it:

• makes sense,
• doesn't change the behavior of the other combinations, and
• doesn't conflict with the flags' current documentation (which would risk surprising users).

That last bullet point is the tough one. The current output of --help says:

  --ip6tables  Enable addition of ip6tables rules (experimental)
  --iptables   Enable addition of iptables rules (default true)

Reasonable users could interpret "Enable addition of iptables rules" to mean either IPv4 rules alone or both IPv4 and IPv6 rules. (I believe most users will interpret it to mean only IPv4 rules, especially after contrasting the --iptables help string with the --ip6tables help string, and after Googling the opaque terms "iptables rules" and "ip6tables rules".)

Proposals

To ground the conversation, here are some specific proposals. Both of these are backwards compatible.

These aren't the only options; they're just the best ones I could immediately think of.

Option 1: --iptables affects both IPv4 and IPv6

  --iptables        Deprecated synonym of --netfilter
  --ip6tables       Deprecated synonym of --netfilter-ipv6
  --netfilter       Allow --netfilter-ipv4 and --netfilter-ipv6 to establish netfilter rules. (default true)
  --netfilter-ipv4  Enable IPv4 netfilter rules.  Ignored when --netfilter=false. (default true)
  --netfilter-ipv6  Enable IPv6 netfilter rules.  Ignored when --netfilter=false. (experimental, default false)

Pros:

• safer: --iptables=false shuts off everything
• matches what some users expect of the --iptables flag (the lack of "4" in the flag name suggests IP in general, not IPv4 specifically)
• amenable to nftables

Cons:

• added cognitive load: three new flags, two deprecations, and a two-stage enable system
• mismatch between what some (most?) users expect of the --iptables flag vs. its actual behavior
• inconsistent with the behavior of the iptables command
• asymmetric behavior: --iptables=false overrides --ip6tables, but --iptables doesn't override --ip6tables=false

Option 2: --iptables only affects IPv4 rules

  --ip6tables  Enable IPv6 netfilter rules. (experimental, default false)
  --iptables   Enable IPv4 netfilter rules. (default true)

Pros:

• simpler
• matches what some (most?) users expect of the --iptables flag
• consistent with the behavior of the iptables command
• the --iptables and --ip6tables flags are orthogonal
• already implemented in Delete erroneous sanity check in setupIPChains #46468 except for improvements to flag documentation

Cons:

• potentially less safe: users might expect --iptables=false to override --ip6tables
• mismatch between what some users expect vs. its actual behavior

[cpuguy83]

I think both these flags are pretty terrible.
--iptables was indeed added as a way to prevent docker from messing with iptables.

Since --iptables=false --ip6tables=true is an error condition today, why not leave it an error condition and implement a better configuration option(s) which we can force users who are wanting to use both these flags into?
Note: when I say "better" I don't mean differently named but still basically the same thing.