Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

firewall-cmd --reload on centos 7.4.1708 removes the DOCKER-USER iptables chain #35043

Closed
aki-k opened this issue Sep 29, 2017 · 10 comments

Comments

Projects
None yet
8 participants
@aki-k
Copy link

commented Sep 29, 2017

Description

firewall-cmd --reload on centos 7.4.1708 removes the DOCKER-USER iptables chain.
The reload keeps the two other chains DOCKER and DOCKER-ISOLATION.

Steps to reproduce the issue:

  1. install CentOS 7
  2. enable firewalld
  3. install docker-ce-17.06.2.ce-1.el7.centos.x86_64 from download.docker.com
  4. start docker engine
  5. iptables -L -n -v > /tmp/iptables_before_firewalld_reload
  6. run firewall-cmd --reload
  7. iptables -L -n -v > /tmp/iptables_after_firewalld_reload
  8. diff -u /tmp/iptables_before_firewalld_reload /tmp/iptables_after_firewalld_reload

Describe the results you received:

DOCKER-USER chain has disappeared.

Describe the results you expected:

DOCKER-USER chain shouldn't have disappeared.

Additional information you deem important (e.g. issue happens only occasionally):

I entered this bug report to CentOS bug tracker at https://bugs.centos.org/view.php?id=13879
but there was no resolution there. They suggested I file a bug report here.

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 20:00:25 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.2-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.13.4-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: deadbeef.internal
ID: 3SP2:GWZC:Q3ZG:USFU:77DC:MPJ3:NHJR:RD7U:MCO6:FV4U:SCAI:TUUW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
I'm testing this on KVM.

@phlegx

This comment has been minimized.

Copy link

commented Oct 3, 2017

Yes this happens to me too on ubuntu when I re-setup my firewall rules with iptable (ansible raw_iptables). I then need to restart docker daemon to get them back completely.

it would be very nice if there would be a docker reload iptables for to get the iptables back into the system iptables. Restarting the docker daemon everytime I re-setup the firewall rules is not a nice option.

@antoinetran

This comment has been minimized.

Copy link

commented Oct 23, 2017

Strange, it does not happen to me with docker-ce-17.06.0, CentOs 7.3. Reloading firewalld has no impact on docker rules. Maybe a regression of docker-ce?

@Yashiroo

This comment has been minimized.

Copy link

commented Jan 7, 2018

I have the same machine as OP. I'm also having this same issue.
I could not see this behavior on a similar machine (CentOS 7.2 and Docker-CE v17).
Is this related to how firewalld manages rules, or to how Docker saves rules to iptables?

@brianbolt

This comment has been minimized.

Copy link

commented Jan 8, 2018

I also have been hunt this down for a while and here is what seems to be working for me:

TLDR first; If you add the DOCKER-USER chain to firewalld so that it is present in iptables before docker starts then you should be able to apply rules.

Longer story; Firewalld and Docker both use iptables to route traffic. Firewalld always flushes iptables rules and only reinstates rules that have been configured with firewalld. Docker, when it starts, adds a number of chains to the iptables that can possibly conflict with your rules. However, Docker does respect a special "DOCKER-USER" chain that you can configure to filter traffic. If the "DOCKER-USER" chain is not present when Docker starts, Docker will add it and allow all connections being passed to it. However, if the DOCKER-USER chain already exists, it will not do anything to it (except add an "ACCEPT ALL rule" to the END of the chain (which won't do anything if you configure the previous rules in the chain cover all traffic cases).

Here is a sequence of commands I have used over and over again to get my firewalld settings correct. My goal was to only allow traffic to http and https from a few ip addresses:

# Removing DOCKER-USER CHAIN (it won't exist at first)
firewall-cmd --permanent --direct --remove-chain ipv4 filter DOCKER-USER

# Flush rules from DOCKER-USER chain (again, these won't exist at first; firewalld seems to remember these even if the chain is gone)
firewall-cmd --permanent --direct --remove-rules ipv4 filter DOCKER-USER

# Add the DOCKER-USER chain to firewalld
firewall-cmd --permanent --direct --add-chain ipv4 filter DOCKER-USER

# Add rules (see comments for details)
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -m comment --comment "This allows docker containers to connect to the outside world"
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -j RETURN -s 172.18.0.0/16 -m comment --comment "allow internal docker communication"
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -p tcp -m multiport --dports https -s 123.456.7.89/32 -j ACCEPT -m comment --comment "my allowed ip address to http and https ports"

#Add as many ip or other rules and then run this command to block all other traffic
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -j REJECT -m comment --comment "reject all other traffic"

# restart the services 
systemctl stop docker
systemctl stop firewalld
systemctl start firewalld
systemctl start docker
@aki-k

This comment has been minimized.

Copy link
Author

commented Jan 12, 2018

@brianbolt

I have a problem with the fact that Docker devs haven't commented on this issue report. Even though your solution works now, it might not work in the future if the CentOS and Docker devs don't agree on the correct method on doing this.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2018

I think the main issue here is that docker does not manager the DOCKER-USER chain.

Perhaps it should re-add the chain when firewalld is reloaded but none of the rules in that chain will be reloaded because these aren't docker's rules.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2018

I think @brianbolt's response is exactly the correct way to deal with this since they are user defined rules.

@aki-k

This comment has been minimized.

Copy link
Author

commented Jan 12, 2018

@cpuguy83

Do you mean the DOCKER-USER chain is created by some CentOS service and not by the docker engine?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2018

@aki-k The DOCKER-USER chain itself is created by docker along with the main jump rule, but that is all. The chain is expected to be managed by users.

I'll open a PR to ensure that the chain and the initial rule is re-created on firewalld reload.

cpuguy83 added a commit to cpuguy83/libnetwork that referenced this issue Jan 12, 2018

Reload DOCKER-USER chain on frewalld reload.
Relates to moby/moby#35043

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2018

opened docker/libnetwork#2053 to ensure the chain is re-created on firewalld reload, but again any user rules will need to be reloaded by the user.

cpuguy83 added a commit to cpuguy83/libnetwork that referenced this issue Jan 12, 2018

Reload DOCKER-USER chain on frewalld reload.
Relates to moby/moby#35043

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.