New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker does not respect previously setup FORWARD rules #29184

Closed
konradkonrad opened this Issue Dec 6, 2016 · 10 comments

Comments

Projects
None yet
6 participants
@konradkonrad
Copy link

konradkonrad commented Dec 6, 2016

Description
I'm trying to restrict outgoing connections from inside a container. From what I gathered, outgoing container traffic is routed via the FORWARD chain of iptables. The problem is, that after restarting the docker service or creating the container, docker will prepend its rules in the FORWARD chain, so my policy is never matched.

Steps to reproduce the issue:

  1. add an iptables rule to drop connections to 10.0.0.0/8 from the br-do bridge device used for the docker network so that iptables --list FORWARD -v reads
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    any     anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do   anywhere             10.0.0.0/8
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
  1. restart docker daemon service docker restart

Describe the results you received:
After restarting the docker daemon, the FORWARD chain will be altered in the following way:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER     all  --  any    br-do  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    br-do  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  br-do  !br-do anywhere             anywhere            
    0     0 ACCEPT     all  --  br-do  br-do  anywhere             anywhere            
    0     0 DROP       all  --  any    any    anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do  anywhere             10.0.0.0/8          
    0     0 ACCEPT     all  --  any    any    anywhere             anywhere             state RELATED,ESTABLISHED

Outgoing traffic on will now be ACCEPTed before matching my intended policy.

Describe the results you expected:
I would have expected that docker adds the FORWARD rules more sensibly, or appends them to the end of the FORWARD chain, so my intended policy stays active.

Additional information you deem important (e.g. issue happens only occasionally):
I understand that it is complex for docker to pick the right behavior here. However, there must be a sensible way to enforce system wide policies without going the DOCKER_OPTS=--iptables=false route.

Output of docker version:

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:15:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:15:17 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 101
 Running: 3
 Paused: 0
 Stopped: 98
Images: 1433
Server Version: 1.12.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1618
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu precise (12.04.5 LTS)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.3 GiB
Name: graph
ID: KCL5:MTXD:FDRD:YXDL:EO73:H2JE:2OZL:W2JX:IDN4:SBBV:CUVP:NTLC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Running on a physical box.

@aboch

This comment has been minimized.

Copy link
Contributor

aboch commented Dec 6, 2016

Have you tried inserting your rule in the DOCKER chain instead ?
That chain is always consulted for packets traversing the FORWARD chain.
And docker should only append in there, IIRC.

@konradkonrad

This comment has been minimized.

Copy link
Author

konradkonrad commented Dec 6, 2016

Thanks @aboch --
I tried it, but to no avail:

  1. manually created the DOCKER chain
Chain DOCKER (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       udp  --  any    br-do  anywhere             10.0.0.0/8          
  1. restarted the docker daemon
    After restart, the DOCKER chain is completely empty.
Chain DOCKER (4 references)
 pkts bytes target     prot opt in     out     source               destination         
@aboch

This comment has been minimized.

Copy link
Contributor

aboch commented Dec 6, 2016

@konradkonrad Yes this is known. Any rule in a docker created chain won't rules won't survive a docker restart.

Ideally we'd need to add a chain where users can add their rules and not flush them at daemon start.
Now I remember this has already been requested. I will try to find the GH issue and link it here.

@aboch

This comment has been minimized.

Copy link
Contributor

aboch commented Dec 6, 2016

The original issue reporting this behavior/feature request in regarding to user defined rules is #24848

@konradkonrad

This comment has been minimized.

Copy link
Author

konradkonrad commented Dec 7, 2016

Any rule in a docker created chain won't rules won't survive a docker restart.

Yeah, this is unfortunate. In my opinion it must be possible to configure firewall policies in way that persists docker operations/restarts.

Ideally we'd need to add a chain where users can add their rules and not flush them at daemon start.

Well either this, or, as a potentially quicker solution, docker could respect certain --comments on rules in the FORWARD chain (I believe this is the only global/default one that is modified by docker). For example, docker could respect rules with the comment DOCKER_BELOW and DOCKER_ABOVE and only insert rules below or above these.

My policy would now become:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    any     anywhere             anywhere             state INVALID
    0     0 DROP       udp  --  any    br-do   anywhere             10.0.0.0/8           
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED  /* DOCKER_BELOW */

And I would docker expect to only insert below the last mention of the comment /* DOCKER_BELOW */.

I don't know how to feel about using comments for any sort of flow control, but since docker is rather intrusive towards the FORWARD chain, this may be quicker to implement and fix the intrusive behavior.

@delfuego

This comment has been minimized.

Copy link

delfuego commented Mar 5, 2017

Is there a hope of fixing this issue? This is... really frustrating, and makes restricting access to ports exposed by containers really really difficult.

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue Mar 8, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-FORWARD-TOP filter chain. This
allows the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby/issues/29184
Fixes moby/moby/issues/23987
Related to moby/moby/issues/24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>
@wenjianhn

This comment has been minimized.

Copy link
Contributor

wenjianhn commented Mar 9, 2017

@delfuego Yes! I have created a pull request to fix it.

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue Mar 13, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue Mar 14, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue May 2, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue May 15, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue May 15, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>

wenjianhn added a commit to wenjianhn/libnetwork that referenced this issue May 16, 2017

Add a filter chain to allow persistent rules
Allow users to configure firewall policies in a way that persists
docker operations/restarts. Docker will not delete or modify any
pre-existing rules from the DOCKER-USER filter chain. This allows
the user to create in advance any rules required to further
restrict access from/to the containers.

Fixes moby/moby#29184
Fixes moby/moby#23987
Related to moby/moby#24848

Signed-off-by: Jacob Wen <jian.w.wen@oracle.com>
@konradkonrad

This comment has been minimized.

Copy link
Author

konradkonrad commented May 17, 2017

woohoo :) 👍

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented May 17, 2017

Reopening for now, because docker/libnetwork#1675 was not yet vendored in this repository, so the code is not in use yet

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jun 7, 2017

Docker 17.06 (through commit 46392f2, which is part of #32981) includes a change to libnetwork (thanks @wenjianhn) that allows setting up a custom "DOCKER-USER" filter chain (see the change in libnetwork upstream: docker/libnetwork#1675), so for those interested; upgrade to 17.06 to get this enhancement

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment