New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker networking fails after iptables service is restarted #12294

Open
markhallm opened this Issue Apr 11, 2015 · 25 comments

Comments

Projects
None yet
@markhallm
Copy link

markhallm commented Apr 11, 2015

Docker networking fails permanently on all new containers after restarting the iptables service. Steps to reproduce on CentOS 7.1 (some output elided for brevity):

$ sudo yum -y install docker iptables-services
$ sudo systemctl start docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms
$ sudo systemctl restart iptables
$ sudo docker run --rm centos bash -c "ping www.docker.com"
ping: unknown host www.docker.com

A workaround to restore networking to containers is to restart the Docker daemon:

$ sudo systemctl restart docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms

What should be fixed in Docker so that reloading iptables rules won't break its networking? This is a major issue because it breaks networking for all containers on the host.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Apr 11, 2015

The advanced networking may contain some information that can help troubleshooting.

Also, could you add the output of;

  • uname -a
  • docker version
  • docker -D info
@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Apr 11, 2015

This is a known issue/limitation. I'm hesitant to call it an issue as I don't think it's docker's problem.
Basically when you run systemctl restart iptables, you're not really restarting iptables as iptables is always on (it's built into the kernel). Instead you're clearing & loading saved rule sets.
Docker adds rules to the iptables chains, so when you clear the chains, the rules docker created are gone.
There is nothing docker can do about this. You just can't wipe the iptables chains.

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Apr 11, 2015

@phemmer There is proposal about firewalld support. WDYT about it?

@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Apr 11, 2015

Great if you're using firewalld :-P
Unfortunately basic iptables is what a lot of people use (myself included).

From the 40,000 foot level, I see only 2 ways to solve this problem:

  1. make docker depend on the iptables service. If iptables is shut down, so is docker. Not great as I'm sure people will not expect docker to go down when iptables does.
    1. Could also have a separate service which depends on the iptables service, so that when it comes up, it pings the docker daemon to tell it to reinstate its rules.
      Maybe as part of the new docker network driver design, the network manager would be a separate process, and it could be stopped/started instead. That way containers would remain running, but the network service wouldn't.
  2. monitor iptables and reinstate the rules if they get deleted. Doesn't look like there's any sort of accounting interface you can use to watch changes, so would have to be polling (and that's a lot of polling if you have a lot of rules). Plus there's still going to be brief gaps where the rules are missing.
  3. Yeah I said 2, but 3: not docker's problem. Just like wiping iptables breaks docker, so would wiping the filesystem, and Docker doesn't prevent that. Just because there's a handy tool for doing so doesn't mean the existence of that tool is something docker should handle. Just throw a warning somewhere in the documentation (if there isn't one already).
@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Apr 12, 2015

I think using firewalld is not very bad option if you're fan of restarting iptables service :)

@gdm85

This comment has been minimized.

Copy link
Contributor

gdm85 commented Apr 13, 2015

I made this utility to cover a similar need: being able to restore sets of iptables rules that affect multiple containers.

@clnperez

This comment has been minimized.

Copy link
Contributor

clnperez commented May 6, 2015

I've seen this on another project I worked on (kimchi: https://github.com/kimchi-project/kimchi/wiki) and we concluded that it's not Kimchi's problem if you wiped the rules it creates on startup. Something in the docs about saving and restoring is the best way to go, IMO.

@virtualfunction

This comment has been minimized.

Copy link

virtualfunction commented Jan 2, 2016

While iptables shouldn't be docker's responsibility if it gets restarted, it would be good if there was a mechanism (maybe there is one hiding, and just needs to be documented to make it clear to users) to save / restore rules docker has created. I'm not really a iptables / sysadmin expert but I guess one could do something with iptables-save and filtering out the DOCKER chain as a basic starting point. (Just noticed in the referenced issue there are instructions, but that saves the whole iptables state - some just want to clear everything except the docker rules)

@chulkilee

This comment has been minimized.

Copy link

chulkilee commented Apr 14, 2016

What about adding a command or mechanism that invoke that network setup phase (updating iptables) without restarting docker daemon?

@michaljrk

This comment has been minimized.

Copy link

michaljrk commented May 12, 2016

Another workaround (?), not elegant at all, but quick.
Put service docker restart in the /etc/init.d/iptables-persistent, in start|restart|reload|force-reload)
section.

@virtualfunction - mainly agree on that.

But I would not say While iptables shouldn't be docker's responsibility

Exactly, if you relay on some other service in OS, it would be nice to assume that user still may use this mechanism for some other purposes (like making host level iptables rules for whatever reason). Then it will be nice to help to compromise this kind of misbehaviour.

+1

@bish0polis

This comment has been minimized.

Copy link

bish0polis commented Sep 30, 2016

sed -i~ '
/^IPTABLES_SAVE_ON_/s/=.*/="yes"/
' /etc/sysconfig/iptables-config

Something I'm not getting, here?

I mean, it's gonna be way harder if you're using chef/etc to scorch and recreate the tables config from scratch each time, but there's a workaround for that (which I was doing at the shop, which led me here) anyway. For most people in the target audience, the tuning above should be all ya need.

@colinmollenhour

This comment has been minimized.

Copy link

colinmollenhour commented Oct 22, 2016

Definitely having a command like "docker network reload-firewall" would be nice so that if you run into a situation you can fix the firewall without having to restart all of your containers..

@taladar

This comment has been minimized.

Copy link

taladar commented Dec 1, 2016

iptables-save is a messy non-solution. Any semi-complex set of rules needs to be written in a more structured way, with comments. You also do not want to save Docker's temporary rules since the saved rules file is usually used on system startup (especially when it results from an unclean shutdown without a save and the saved rules file is a little older), when half of them are obsolete because the containers they belonged to no longer exist.

Docker simply needs a command to recreate its rules without restarting docker itself. It does not have to determine automatically when that command has to be called, it is easy enough to add it to any script flushing and recreating firewall rules. Honestly, I can't see how anyone can run Docker on a production server with any sort of availability requirement without a way to use iptables for other things.

@gdm85

This comment has been minimized.

Copy link
Contributor

gdm85 commented Dec 2, 2016

Meanwhile, there's been some progress in go-libiptc, you might want to give a look for a possible lower-level integration. Has anybody tried my docker-fw with most recent Docker?

@qwIvan

This comment has been minimized.

Copy link

qwIvan commented Jul 25, 2017

I want to recreate iptables rules without restart dockerd

@taladar

This comment has been minimized.

Copy link

taladar commented Jul 25, 2017

The fact that this ticket is still open is one of the things I regularly point to when people ask why I consider Docker far from production ready (the others include the lack of proper garbage collection).

Quite frankly the whole Docker container hype is a bit of a bad joke while tickets like this one are completely ignored.

@strootman

This comment has been minimized.

Copy link

strootman commented Jul 25, 2017

@taladar All of GH is running on Kubernetes! IMO, it's well past hype.

@taladar

This comment has been minimized.

Copy link

taladar commented Jul 25, 2017

It works for anything large enough that you can just take a host where Docker breaks out of your load balancer and fix it while the rest keeps on running but if you actually need your hosts to work reliably without a few hours of unplanned downtime every once in a while Docker still isn't there yet.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jul 26, 2017

It would be inappropriate for Docker to re-insert rules automatically after they have been removed from iptables.

Providing an API to re-init iptables rules is something to look at, but it's not obvious (to me) what this API would look like.
It could be supported on "SIGUSR1", but then that's really starting to overload that function.

@bish0polis

This comment has been minimized.

Copy link

bish0polis commented Jul 26, 2017

@taladar

This comment has been minimized.

Copy link

taladar commented Jul 26, 2017

You can not save and restore Docker iptables rules externally unless you have only static, long-running Docker containers with exposed ports.

What Docker really just needs is the same thing most other, similar software does in this case. Call a configurable script on each event (startup, container start, container stop, shutdown) and pass all the relevant data to the script as parameters.

@paklids

This comment has been minimized.

Copy link

paklids commented Jun 8, 2018

I have a slightly different use case and have been frustrated there isn't a workaround. I'm pushing out very specific iptables rules using ansible and those remove the docker created rules. So I wrote some shell scripts (leveraging netfilter-persistent) that make a backup of the running rules, allow me to push my ansible rules, then restore only the docker specific ones. Its a bit of a hack but gets me closer to my goal. If these might be useful for you then you can pull them down at:

https://github.com/paklids/docker-iptables-restore

@ahmadalli

This comment has been minimized.

Copy link

ahmadalli commented Jul 10, 2018

kube-proxy is doing the same but also reattach itself to iptables after reload

@hudsantos

This comment has been minimized.

Copy link

hudsantos commented Oct 25, 2018

This helped me a lot with this issue: https://unrouted.io/2017/08/15/docker-firewall/
Baiscally doing things with iptables-restore -n to avoid flushing existing rules.. Docker rules untouched.

@taladar

This comment has been minimized.

Copy link

taladar commented Oct 25, 2018

As a workaround for Docker's inability to play nice with the rest of the system we are now just isolating Docker in its own named network namespace using

https://github.com/Jamesits/systemd-named-netns

That way Docker has its own iptables rules and its own IP and we can finally modify the regular iptables rules freely without requiring constant Docker restarts that take ages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment