docker does not respect previously setup FORWARD rules

[konradkonrad]

Description
I'm trying to restrict outgoing connections from inside a container. From what I gathered, outgoing container traffic is routed via the FORWARD chain of iptables. The problem is, that after restarting the docker service or creating the container, docker will prepend its rules in the FORWARD chain, so my policy is never matched.

Steps to reproduce the issue:

1. add an iptables rule to drop connections to 10.0.0.0/8 from the br-do bridge device used for the docker network so that iptables --list FORWARD -v reads

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    any     anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do   anywhere             10.0.0.0/8
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED

2. restart docker daemon service docker restart

Describe the results you received:
After restarting the docker daemon, the FORWARD chain will be altered in the following way:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER     all  --  any    br-do  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    br-do  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  br-do  !br-do anywhere             anywhere            
    0     0 ACCEPT     all  --  br-do  br-do  anywhere             anywhere            
    0     0 DROP       all  --  any    any    anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do  anywhere             10.0.0.0/8          
    0     0 ACCEPT     all  --  any    any    anywhere             anywhere             state RELATED,ESTABLISHED

Outgoing traffic on will now be ACCEPTed before matching my intended policy.

Describe the results you expected:
I would have expected that docker adds the FORWARD rules more sensibly, or appends them to the end of the FORWARD chain, so my intended policy stays active.

Additional information you deem important (e.g. issue happens only occasionally):
I understand that it is complex for docker to pick the right behavior here. However, there must be a sensible way to enforce system wide policies without going the DOCKER_OPTS=--iptables=false route.

Output of docker version:

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:15:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:15:17 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 101
 Running: 3
 Paused: 0
 Stopped: 98
Images: 1433
Server Version: 1.12.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1618
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu precise (12.04.5 LTS)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.3 GiB
Name: graph
ID: KCL5:MTXD:FDRD:YXDL:EO73:H2JE:2OZL:W2JX:IDN4:SBBV:CUVP:NTLC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Running on a physical box.

[aboch]

Have you tried inserting your rule in the DOCKER chain instead ?
That chain is always consulted for packets traversing the FORWARD chain.
And docker should only append in there, IIRC.

[konradkonrad]

Thanks @aboch --
I tried it, but to no avail:

1. manually created the DOCKER chain

Chain DOCKER (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       udp  --  any    br-do  anywhere             10.0.0.0/8          

2. restarted the docker daemon
   After restart, the DOCKER chain is completely empty.

Chain DOCKER (4 references)
 pkts bytes target     prot opt in     out     source               destination         

[aboch]

@konradkonrad Yes this is known. Any rule in a docker created chain won't rules won't survive a docker restart.

Ideally we'd need to add a chain where users can add their rules and not flush them at daemon start.
Now I remember this has already been requested. I will try to find the GH issue and link it here.

[aboch]

The original issue reporting this behavior/feature request in regarding to user defined rules is #24848

[konradkonrad]

Any rule in a docker created chain won't rules won't survive a docker restart.

Yeah, this is unfortunate. In my opinion it must be possible to configure firewall policies in way that persists docker operations/restarts.

Ideally we'd need to add a chain where users can add their rules and not flush them at daemon start.

Well either this, or, as a potentially quicker solution, docker could respect certain --comments on rules in the FORWARD chain (I believe this is the only global/default one that is modified by docker). For example, docker could respect rules with the comment DOCKER_BELOW and DOCKER_ABOVE and only insert rules below or above these.

My policy would now become:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    any     anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do   anywhere             10.0.0.0/8           
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED  /* DOCKER_BELOW */

And I would docker expect to only insert below the last mention of the comment /* DOCKER_BELOW */.

I don't know how to feel about using comments for any sort of flow control, but since docker is rather intrusive towards the FORWARD chain, this may be quicker to implement and fix the intrusive behavior.

[delfuego]

Is there a hope of fixing this issue? This is... really frustrating, and makes restricting access to ports exposed by containers really really difficult.

[wenjianhn]

@delfuego Yes! I have created a pull request to fix it.

[konradkonrad]

woohoo :) 👍

[thaJeztah]

Reopening for now, because moby/libnetwork#1675 was not yet vendored in this repository, so the code is not in use yet

[thaJeztah]

Docker 17.06 (through commit 46392f2, which is part of #32981) includes a change to libnetwork (thanks @wenjianhn) that allows setting up a custom "DOCKER-USER" filter chain (see the change in libnetwork upstream: moby/libnetwork#1675), so for those interested; upgrade to 17.06 to get this enhancement