Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker vs. firewalld on CentOS 7 #16137

Closed
stephanadler opened this issue Sep 8, 2015 · 96 comments
Closed

Docker vs. firewalld on CentOS 7 #16137

stephanadler opened this issue Sep 8, 2015 · 96 comments

Comments

@stephanadler
Copy link

@stephanadler stephanadler commented Sep 8, 2015

Hi.
In https://docs.docker.com/v1.6/installation/centos/#installing-docker-centos-7 it is documented that docker can be run together with firewalld if you respect the order in which services are started, however I seem to have problems getting that to run. As far as the documentation says I would assume that the following would result in a clean state.

[root@App1 ~]# systemctl stop docker
[root@App1 ~]# systemctl stop firewalld
[root@App1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
[root@App1 ~]# systemctl start firewalld
[root@App1 ~]# systemctl start docker

Everything went fine up to now, I also can run containers, they have network connectivity etc.. The thing which does not seem to work is inter container communication. While debugging this, I came across the following lines in /var/log/firewalld, which are a direct result of the [root@App1 ~]# systemctl start docker command above:

2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.

My current thesis is, that a running firewalld causes a kind of a race condition which causes that the docker chain is not yet existing when docker already tries to add rules to it. When researching that issue I found other people having trouble with CentOS 7 and Docker, but no one so far described that behavior.

docker version:
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.centos.x86_64
Go version (client): go1.4.2
Git commit (client): 3043001/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Package Version (server): docker-1.7.1-108.el7.centos.x86_64
Go version (server): go1.4.2
Git commit (server): 3043001/1.7.1
OS/Arch (server): linux/amd64
docker info:
Containers: 8
Images: 155
Storage Driver: devicemapper
 Pool Name: docker-253:0-3146366-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.95 GB
 Data Space Total: 107.4 GB
 Data Space Available: 40.16 GB
 Metadata Space Used: 9.466 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 12
Total Memory: 31.22 GiB
Name: App1
ID: SHSS:NWHQ:V3C7:KUCO:VUH6:4VH3:AOYV:N3TN:YTLI:7M5F:OKSZ:6FWY
uname -a
Linux USCL-HDApp1 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Sep 8, 2015

ping @mavenugo has this come up before i think it may be a dup idk

@smuthali
Copy link

@smuthali smuthali commented Sep 8, 2015

@stephanadler
@mavenugo

I have the same/similar issues as mentioned by @stephanadler .
Attempted both with docker upstream package (docker-engine 1.8.1) and CentOS provided packages (docker 1.7.1) on CentOS 7.1 - I see unpredictable behavior with docker-engine/docker client startup. This happens consistently with firewalld running and sporadically with firewalld stopped/disabled. I am happy to provide addition details, but I do not want to clutter this post.

@Djelibeybi
Copy link
Contributor

@Djelibeybi Djelibeybi commented Sep 8, 2015

The same issue exists (unsurprisingly) on Oracle Linux 7 as well.

@ystyle
Copy link

@ystyle ystyle commented Sep 24, 2015

i had the same issue , Through this article solve this problem (the article Chinese not English,who can translate that )
http://www.lxy520.net/2015/09/24/centos-7-docker-qi-dong-bao/

@fud
Copy link

@fud fud commented Sep 29, 2015

@stephanadler I'm having this exact issue, was there any solution to this?

@stephanadler
Copy link
Author

@stephanadler stephanadler commented Sep 29, 2015

@fud right now I am running the system with disabled firewalld. That's quite ugly, but up to now I tend to think this is a real bug and therefore everything which can be used as workarround is somewhat ugly. The chinese article posted by @ystyle seems to indicate to let the firewall management (in our case firewalld) create the DOCKER chain. I did not yet play arround with that option, probably will in the next days.

@smuthali
Copy link

@smuthali smuthali commented Sep 29, 2015

@fud @stephanadler - even with firewalld disabled I have trouble with docker-engine starting. Apparently docker-engine 1.9 will address this issue.

@fud
Copy link

@fud fud commented Sep 29, 2015

@stefanberger @smuthali Thanks, I will await for a solution from docker.

@AndreaGiardini
Copy link

@AndreaGiardini AndreaGiardini commented Nov 9, 2015

This is still broken in CentOS 7

[vagrant@myprecise ~]$ uname -a
Linux myprecise.box 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@myprecise ~]$ sudo docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Package Version: docker-1.8.2-7.el7.centos.x86_64
 Go version:   go1.4.2
 Git commit:   bb472f0/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   bb472f0/1.8.2
 Built:        
 OS/Arch:      linux/amd64
[vagrant@myprecise ~]$ sudo service firewalld status -l
Redirecting to /bin/systemctl status  -l firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Mon 2015-11-09 16:20:28 UTC; 2min 15s ago
 Main PID: 22098 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─22098 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 11, 2015

@AndreaGiardini have you tested with docker 1.9 as well?

@stephanadler
Copy link
Author

@stephanadler stephanadler commented Nov 12, 2015

@thaJeztah I just installed a fresh CentOS 7 and installed the current docker release:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.

@AndreaGiardini
Copy link

@AndreaGiardini AndreaGiardini commented Nov 12, 2015

I confirm as well

@AndreaGiardini
Copy link

@AndreaGiardini AndreaGiardini commented Nov 16, 2015

[vagrant@myprecise yum.repos.d]$ sudo service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Mon 2015-11-16 14:57:31 UTC; 1s ago
 Main PID: 9963 (firewalld)
   CGroup: /system.slice/firewalld.service
           ├─ 9963 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
           └─10319 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Nov 16 14:57:28 myprecise.box systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 16 14:57:31 myprecise.box systemd[1]: Started firewalld - dynamic firewall daemon.
[vagrant@myprecise yum.repos.d]$ sudo service docker start
Redirecting to /bin/systemctl start  docker.service
[vagrant@myprecise yum.repos.d]$ sudo service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Mon 2015-11-16 14:57:31 UTC; 8s ago
 Main PID: 9963 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─9963 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/tar...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: i...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEP...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACC...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctst...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DO...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: ...that name.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@myprecise yum.repos.d]$ sudo service docker status
Redirecting to /bin/systemctl status  docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
   Active: active (running) since Mon 2015-11-16 14:57:38 UTC; 19s ago
     Docs: https://docs.docker.com
 Main PID: 10356 (docker)
   CGroup: /system.slice/docker.service
           └─10356 /usr/bin/docker daemon -H fd://

Nov 16 14:57:36 myprecise.box docker[10356]: time="2015-11-16T14:57:36.744523806Z" level=warning msg="Usage of loopback devices is strongly discouraged for p... section."
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.062164989Z" level=info msg="API listen on /var/run/docker.sock"
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.103601717Z" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.137470288Z" level=info msg="Firewalld running: true"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.145007406Z" level=info msg="Default bridge (docker0) is assigned with an IP address 17...P address"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304092328Z" level=info msg="Loading containers: start."
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304312664Z" level=info msg="Loading containers: done."
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304336576Z" level=info msg="Daemon has completed initialization"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304360310Z" level=info msg="Docker daemon" commit=76d6bc9 execdriver=native-0.2 graphd...sion=1.9.0
Nov 16 14:57:38 myprecise.box systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@myprecise yum.repos.d]$ sudo docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 16, 2015

ping @mavenugo @mrjana could you have a look at this one? some people running into this on 1.9.0 (see the discussion above)

@InfoSec812
Copy link

@InfoSec812 InfoSec812 commented Nov 25, 2015

We are also seeing a similar issue with Docker on CentOS 7 with firewalld running. Intermittently we will see networking issues with container either being inaccessible or unable to communicate with external services. I will try to get more details the next time the issue arises.

@mavenugo
Copy link
Contributor

@mavenugo mavenugo commented Nov 25, 2015

@jfrazelle @thaJeztah we had issues with selinux package in 1.8.x which caused issues in docker to firewalld interaction. But this seems to be a different issue. It is not very clear what is the actual issue seen in 1.9.0 (other than the error messages seen in the firewalld service).

@mrjana the only thing i could think of that is different in 1.9.0 is the way we handle the docker0 bridge cleanup and restart during daemon restarts. Do you think this could cause any problems with firewalld ?

@mavenugo
Copy link
Contributor

@mavenugo mavenugo commented Nov 25, 2015

@aboch this could be a bridge driver specific initializing issue. can you PTAL ?

@filethis-dev-site
Copy link

@filethis-dev-site filethis-dev-site commented Nov 30, 2015

I’m not certain, but it sounds like I have the same problem as what I read about above.

I have a fresh CentOS 7 install, turned off SELinux, turned off firewalld, and when I try to start my Docker container, I get:

Error response from daemon: Cannot start container FOO: failed to create endpoint BAR on network bridge: COMMAND_FAILED: '/sbin/iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 9292 -j DNAT --to-destination 172.17.0.2:9292 ! -i docker0' failed: iptables: No chain/target/match by that name.

Version:

$ docker --version
Docker version 1.9.1, build a34a1d5

If it helps, this is what iptables shows me:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
@ystyle
Copy link

@ystyle ystyle commented Nov 30, 2015

@filethis-dev-site

iptables-save > /etc/sysconfig/iptables

modify this file like this

*nat
:PREROUTING ACCEPT [27:11935]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [598:57368]
:POSTROUTING ACCEPT [591:57092]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
COMMIT  
# Completed on Sun Sep 20 17:35:31 2015
# Generated by iptables-save v1.4.21 on Sun Sep 20 17:35:31 2015
*filter
:INPUT ACCEPT [139291:461018923]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [127386:5251162]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
COMMIT  
# Completed on Sun Sep 20 17:35:31 2015

restart iptables
systemctl restart iptables.service

@LeoShi
Copy link

@LeoShi LeoShi commented Nov 30, 2015

@ystyle Thank you for your solution, but I failed in last step:

[lei@tang workspace]$ sudo systemctl restart iptables.service
Failed to issue method call: Unit iptables.service failed to load: No such file or directory.

and I solved this issue by this http://stackoverflow.com/questions/24756240/how-can-i-use-iptables-on-centos-7

@SAKUJ0
Copy link

@SAKUJ0 SAKUJ0 commented Feb 1, 2016

Would you gents, please, mind adding a note to the documentation about how Docker + CentOS/RHEL in its default configuration is a very bad idea right now?

A lot of things suggest that el7 is a pretty good platform for a host that is supposed to do nothing apart from docker. (advertizing from RH, documentation from RH, being #2 after ubuntu in the docker documentation, RH deprecating LXC in favor of docker, the mere fact that RH "supports" docker, as it does not strive to support more than is good).

It is not a good platform.

This is reproducible on all our CentOS hosts. All our Ubuntu and Arch Linux hosts are still standing strong with docker. It's all deployed using ansible. This does cause actual issues (at the very least after a reboot without any docker autostart.)

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Feb 1, 2016

@SAKUJ0 which version of docker are you installing? The Red Hat maintained packages, or the packages from https://yum.dockerproject.org? Could you explain what you mean with "at the very least after a reboot without any docker autostart"?

@SAKUJ0
Copy link

@SAKUJ0 SAKUJ0 commented Feb 1, 2016

Edit It does not appear to be a MSS issue but a Docker with CentOS 7 related issue. I will try out the CentOS package (despite people saying it would not change anything) and mess around with some more containers. But without masquerading involved, those kind of issues tend to always be firewall(read: iptables) related on the client that can't be reached (after all the entire rest of the network can be reached). Locally it works.

@thaJeztah

To answer your question: We use our (read the official docker) yum repo following the instructions from "Installing Docker" in our official documentation.

It's a bit messy. I have dockera and dockerb. dockera was running docker for now ~ 4 months or so. This Sunday, the web application had issues (potentially unrelated with docker but instead related with Fragment and MSS).

After the reboot I moved all the volumes from dockera to dockerb and re-deployed. They are symmetrical hosts, but I keep dockerb perfectly up-to-date and dockera I keep quite stagnant (it's local only).

Then I re-installed dockera (again all using ansible) and it solved the issues. Until I rebooted. What both hosts have in common are the error messages described in this thread (and it does not look like with those chains missing forwarding can work decently).

Here are the logs.

http://hastebin.com/wavobicexi.vbs
http://hastebin.com/qobopuqaqa.vbs

(it's a pastebin without all the bloat).

While going through them, I think I was dead wrong when I said "actual issues". It seems they are just error messages with no other things related and the VPN issues were just coincidence.

Though, quite reproducible for me. Host b was naked and fully updated and running a functional DNS server only for the longest amount of time. You can see the issue on host b only once. That is because I never rebooted it.

This could be a bit instructive as I could grab out the ansible playbooks used to deploy the docker and firewalld portions.

@binarytemple
Copy link

@binarytemple binarytemple commented Feb 9, 2016

Just installed Docker 1.10 (upgrade but I destroyed all the existing configuration/LVM/etc).

# docker version        
Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

Got the following messages from tail -f /var/log/firewalld on service docker start.

2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: Too many links.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: Too many links.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.

I can appreciate there must be quite some work in making docker play nice with firewalld (I'm perfectly happy with iptables command myself) - but it would be nice to put a big warning on the page indicating that the two services (docker and firewalld) don't play together particularly well.

@davclark
Copy link

@davclark davclark commented Feb 9, 2016

+1 to @binarytemple's doc suggestions.

To be clear - the only functional solution (AFAIK) on CentOS now is to use docker 1.8 from CentOS extras (which still generates warnings, but firewalld works - not sure if docker networking is fully functional) OR mask firewalld and use iptables?

@willseward
Copy link

@willseward willseward commented Feb 9, 2016

+1 for @binarytemple's documentation

@J77J
Copy link

@J77J J77J commented Aug 17, 2017

@xcellardoor ok, that'll be a no then...

looks like I'll have to switch out firewalld for iptables in that case.

Thanks!

@MaciejKucia
Copy link

@MaciejKucia MaciejKucia commented Sep 1, 2017

Not sure if this is fixed or not but one cannot have too many logs 😃

docker --version
Docker version 1.10.3, build 79ebcd8-unsupported
journalctl  _SYSTEMD_UNIT=firewalld.service --no-pager
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -I DOCKER -i docker0 -j RETURN' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2005 -j DNAT --to-destination 172.17.0.7:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2300 -j DNAT --to-destination 172.17.0.8:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2110 -j DNAT --to-destination 172.17.0.9:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2000 -j DNAT --to-destination 172.17.0.2:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --out-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2001 -j DNAT --to-destination 172.17.0.3:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2002 -j DNAT --to-destination 172.17.0.4:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2003 -j DNAT --to-destination 172.17.0.5:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2004 -j DNAT --to-destination 172.17.0.6:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete OUTPUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2300 -j DNAT --to-destination 172.17.0.8:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2003 -j DNAT --to-destination 172.17.0.5:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2000 -j DNAT --to-destination 172.17.0.2:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2004 -j DNAT --to-destination 172.17.0.6:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2005 -j DNAT --to-destination 172.17.0.7:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2110 -j DNAT --to-destination 172.17.0.9:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:03 PC4 firewalld[11649]: 2017-09-01 12:46:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2002 -j DNAT --to-destination 172.17.0.4:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:04 PC4 firewalld[11649]: 2017-09-01 12:46:04 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2001 -j DNAT --to-destination 172.17.0.3:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                      
                                      Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                      
                                      Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                      
                                      Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
@reduardo7
Copy link

@reduardo7 reduardo7 commented Dec 1, 2017

I use an auxiliar script like next:

docker-start.sh

#!/usr/bin/env bash

set -e
set -x

docker-compose up -d
sleep 5

# #Fix1: Fix "iptable service restart" error

echo 'Fix "iptable service restart" error'
echo 'https://github.com/moby/moby/issues/16137#issuecomment-160505686'

for container_id in $(docker ps --filter='ancestor=reduardo7/my-image' -q)
  do
    docker exec $container_id sh -c 'iptables-save > /etc/sysconfig/iptables'
  done

# End #Fix1

echo Done
@kwojcicki
Copy link
Contributor

@kwojcicki kwojcicki commented Dec 5, 2017

Hey @tiangolo I just tried to do the following on a fresh CentOS box (7.4.1708)

nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
nmcli connection modify docker0 connection.zone trusted
systemctl restart docker.service

Following this comment: #16137 (comment)
And i still have issues where my docker containers cannot connect to one another using the host IP.

Despite the fact that

firewall-cmd --get-zone-of-interface=docker0

and

nmcli connection show docker0 | grep zone

return trusted

In addition

cat /etc/sysconfig/network-scripts/ifcfg-docker0
DEVICE=docker0
STP=no
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=172.17.0.1
PREFIX=16
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV4_DNS_PRIORITY=100
IPV6INIT=no
NAME=docker0
ONBOOT=no
ZONE=trusted


sudo cat /etc/firewalld/zones/trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <interface name="docker0"/>
</zone>
@kwojcicki
Copy link
Contributor

@kwojcicki kwojcicki commented Dec 8, 2017

Replying to my previous comment. The issue was our docker-compose specified another network ie

networks:
  default:

Therefore all our containers were not on the docker0 bridge but on a randomly generated bridge.

To fix this restructured our docker-compose.yml

networks:
  default:
  driver_opts:
  com.docker.network.bridge.name: "dockernet"

Next ran

firewall-cmd --permanent --zone=trusted --change-interface=dockernet

And worked like a charm

@PMarci
Copy link

@PMarci PMarci commented Jan 23, 2018

A combination of @kwojcicki's and @tiangolo's comments solved my issue. The published addresses of my containers were unreachable through the host IP from inside of other containers. Disabling firewalld was not an option, as we needed its NAT routing to access external resources.

@Angelinsky7
Copy link

@Angelinsky7 Angelinsky7 commented Feb 3, 2018

@tiangolo thank you so much!!!! You Rock !

hellfirehd added a commit to provisiondata/pdsiss that referenced this issue Apr 28, 2018
@antoinetran
Copy link

@antoinetran antoinetran commented Jun 4, 2018

@tiangolo : our team found this issue and applied in our environment: CentOs 7.4.1708 / docker-ce-17.12.1.ce-1.el7.centos.x86_64 / Swarm classic 1.2.8.
Do you think this workaround still applies now or is this deprecated? This is strange we have to work around firewall rules as this is something docker should do, isn't it? Thank you.

@tiangolo
Copy link

@tiangolo tiangolo commented Jun 5, 2018

@antoinetran yes, Docker should handle firewall rules, but this was a bug in RedHat, inherited by CentOS, it was not misbehavior of Docker but a bug in RedHat.

It was supposedly fixed in RedHat and there was supposedly an update / fix in CentOS. You can read the last comments in the issue in Firewalld: firewalld/firewalld#195

I don't know if that fix works, nor if all the description and workaround still applies because I don't use RedHat/CentOS very frequently.

But by recent comments of this year by @PMarci and @Angelinsky7 , it seems it still applies.

@antoinetran
Copy link

@antoinetran antoinetran commented Jun 5, 2018

@PMarci and @Angelinsky7 : can you tell us your CentOs/RedHat version at the time of the patch? The related RedHat issue here says this is fixed since 7.3.

@Angelinsky7
Copy link

@Angelinsky7 Angelinsky7 commented Jun 5, 2018

@antoinetran CentOS Linux release 7.4.1708 (Core)

@ChristianCiach
Copy link

@ChristianCiach ChristianCiach commented Jun 5, 2018

Today I updated a CentOS 7 system by using "yum update". This also updated Docker-CE. System is now "CentOS Linux release 7.5.1804 (Core)" and Docker is "18.05.0.ce-3.el7.centos". The previous system update was done a few weeks ago.

Now DNS resolution does not work anymore from inside docker containers. Maybe this is related to this issue?

EDIT: Deleting /var/lib/docker/network/files/ fixed my issue. Maybe it's unrelated to this issue after all.

@kwojcicki
Copy link
Contributor

@kwojcicki kwojcicki commented Jun 5, 2018

@Angelinsky7 our team is also using centos 7.4.1708 and using my fix above it all works fine.

@PMarci
Copy link

@PMarci PMarci commented Jun 5, 2018

@antoinetran Unfortunately I'm unable to check it as it's on a customer's on-premise system, and I don't have access right now.

@antoinetran
Copy link

@antoinetran antoinetran commented Jun 5, 2018

Ok thank you all. I added that info in the related RedHat issue here . I hope they will reopen it.

@Chaz6
Copy link

@Chaz6 Chaz6 commented Apr 11, 2019

I have this issue on CentOS 7.6.1810. Docker version is 1.13.1, build b2f74b2/1.13.1.

Apr 11 09:25:31 vps-4 dockerd-current: time="2019-04-11T09:25:31.195015109+01:00" level=info msg="Firewalld running: true"
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j D
OCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptab
les v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule
 exist in that chain?).
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exi
st in that chain?).
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name
.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name
.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match
 by that name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match
 by that name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by tha
t name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/ma
tch by that name.
@deepaklor
Copy link

@deepaklor deepaklor commented Apr 18, 2019

in a NUTSHELL,

The below commands might be needed when the firewalld is enabled on the centos 7, is that correct? unless using different interface for docker.

nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
nmcli connection modify docker0 connection.zone trusted
systemctl restart docker.service

@robertoschwald
Copy link

@robertoschwald robertoschwald commented Jun 13, 2019

While adding the docker0 interface to the trusted zone might be a solution for one who wants to expose the Docker ports to public, but it is not for use cases where you need to protect the port(s) and only allow named IPs.

Solution is to let firewalld create the DOCKER-USER chain and apply rules to it.
See https://roosbertl.blogspot.com/2019/06/securing-docker-ports-with-firewalld.html

@pasikarkkainen
Copy link

@pasikarkkainen pasikarkkainen commented Jun 13, 2019

Note that the DOCKER-USER url ( https://roosbertl.blogspot.com/2019/06/securing-docker-ports-with-firewalld.html ) only works for docker-ce, not for native docker in rhel7/centos7.

There's a RFE about DOCKER-USER support in the rhel7 native docker 1.13: https://bugzilla.redhat.com/show_bug.cgi?id=1678883

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.