Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading docker 1.13 on nodes causes outbound container traffic to stop working #40182

Closed
colemickens opened this issue Jan 19, 2017 · 46 comments

Comments

@colemickens
Copy link
Contributor

commented Jan 19, 2017

Kubernetes version (use kubectl version): v1.4.6, v1.5.1, likely many versions

Environment:

  • Cloud provider or hardware configuration: Azure / Azure Container Service
  • OS (e.g. from /etc/os-release): Ubuntu Xenial
  • Kernel (e.g. uname -a): latest 16.04-LTS kernel
  • Install tools: Cloud-Init + hyperkube
  • Others:

Configuration Details:

  • kubelet runs in a container
  • master services run as static manifests
  • kube-addon-manager runs as a static manifest
  • kube-proxy runs in iptables mode via a daemonset

What happened:
After upgrading to docker 1.13.0 on the nodes, outbound container traffic stops working

What you expected to happen:
Outbound container traffic to work (aka, I can hit the internet and service ips from inside the container)

How to reproduce it (as minimally and precisely as possible):
Deploy an ACS Kubernets cluster. If the workaround has rolled out, then force upgrade docker to 1.13 (you'll have to remove a pin we're setting in /etc/apt/preferences.d).

Unclear if this repros on other configurations right now.

Anything else do we need to know:

No, I just don't know where/how to best troubleshoot this.

@kargakis

This comment has been minimized.

Copy link
Member

commented Jan 20, 2017

@kubernetes/sig-node-misc

@dkerwin

This comment has been minimized.

Copy link

commented Jan 23, 2017

Can confirm the problem with k8s 1.4.7 & docker 1.13 on debian jessie. kubelet managed by systemd

@colemickens

This comment has been minimized.

Copy link
Contributor Author

commented Jan 24, 2017

Since the team @kargakis tagged here is no longer a team... cc: @kubernetes/sig-node-bugs

@bboreham

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2017

Docker 1.13 changed the default iptables forwarding policy to DROP, which has effects like this.

You can change the policy to ACCEPT (which it was in Docker 1.12 and before) by running:

sudo iptables -P FORWARD ACCEPT

on every node. You need to run this in the host network namespace, not inside a pod namespace.

@Misteur-Z

This comment has been minimized.

Copy link

commented Jan 31, 2017

sudo iptables -P FORWARD ACCEPT

Tested out and working

Environment:

  • Cloud provider: AWS
  • OS: Ubuntu Server 16.04 LTS
  • Kernel: 4.4.0-59
  • Kubernetes: 1.5.2
  • Docker: 1.13
  • Network plugin: Weave 1.8.2
  • Network: VPC with NAT gateway, internet gateway, public & private subnets
@feiskyer

This comment has been minimized.

Copy link
Member

commented Feb 1, 2017

Docker 1.13 changed the default iptables forwarding policy to DROP, which has effects like this.

Could someone explain why docker defaulting to DROP? Does this mean containers of docker v1.13 can't connect outside by default?

@bboreham

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2017

@feiskyer generally the Linux default is to have IP forwarding off.
Docker used to turn it on across the board, which was (a) unnecessary and (b) a security issue. 1.13 removed this issue.

Docker add two specific rules which allow traffic off their bridge, and replies to come back:

-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

CNI providers which do not use the docker0 bridge need to make similar provision.

@bboreham

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2017

@colemickens can you clarify which network plugin you are using - is it kubenet?

@Karimerto

This comment has been minimized.

Copy link

commented Feb 1, 2017

This thread sure was a lifesaver, though sadly I found it about 14 hours too late.. I managed to wipe my entire cluster and reinstall everything, with the same issue still persisting. I was about to lose my mind trying to figure out why half of my original cluster was working and the other wasn't. Those which didn't work were installed and added later with docker 1.13, so this explains everything.

Now I've got everything up and running again!

Thanks again for this 👍

@jbeda

This comment has been minimized.

Copy link
Contributor

commented Feb 2, 2017

The docker change that caused this: moby/moby#28257

@colemickens

This comment has been minimized.

Copy link
Contributor Author

commented Feb 2, 2017

(@bboreham Yes, it was kubenet.)

@jbeda

This comment has been minimized.

Copy link
Contributor

commented Feb 2, 2017

The way that services work with iptables mean that we will be initiating connections in to the bridge. As such, we need to be more permissive than the docker0 lines above as we won't already have a connection for incoming connections. Here is what I had to do (my bridge is called cni0):

iptables -A FORWARD -i cni0 -j ACCEPT
iptables -A FORWARD -o cni0 -j ACCEPT

This says: forward stuff both in and out to cni0 regardless.

@antoineco

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2017

It used to be recommended to start Docker with --iptables=false and --ip-masq=false in a Kubernetes deployment, would it be a viable approach to ensure these are effectively disabled using a kubelet flag?

ref. https://kubernetes.io/docs/getting-started-guides/scratch/#docker

@bboreham

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2017

Personally I would be happier with the minimum required to make Kubernetes work, rather than blanket turning on IP forwarding from anywhere to anywhere.

Also, it seems to me that specifying --iptables=false while leaving --ip-forward defaulted to true is a very obscure way to arrive at your desired result.

@aaron-trout

This comment has been minimized.

Copy link

commented Feb 10, 2017

This also appears to stop NodePort services from working. (Thanks to @bboreham for the workaround!)

@jbeda

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2017

sorry -- accidental close

@jbeda jbeda reopened this Mar 2, 2017
@caseydavenport

This comment has been minimized.

Copy link
Member

commented Oct 4, 2017

@euank I'm not entirely convinced this is the responsibility of CNI plugins. It's discussed above, and some plugins do include workarounds, but manipulating the host to allow forwarding of traffic feels squarely outside of a CNI plugin's responsibilities as I understand them.

I think we still need a solution for this in Kubernetes.

CC @tmjd

@squeed

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2017

We're adding a "allow this interface" chained plugin in to the CNI repository. It's a clean solution to this problem.

containernetworking/plugins#75

@digglife

This comment has been minimized.

Copy link

commented Nov 9, 2017

@bboreham it works. Thank you!

@feiskyer

This comment has been minimized.

Copy link
Member

commented Nov 10, 2017

@squeed iptables-allow is cool. We should document this clearly when the plugin is ready.

@sathieu

This comment has been minimized.

Copy link

commented Nov 22, 2017

#52569 looks to be the proper fix for this (since v1.9.0-alpha.2).

@cmluciano

This comment has been minimized.

Copy link
Member

commented Nov 22, 2017

closed with #52569

@cmluciano cmluciano closed this Nov 22, 2017
justinsb added a commit to justinsb/kops that referenced this issue Dec 1, 2017
Docker 1.13 changed how it set up iptables in a way that broke
forwarding.

We previously got away with it because we set the ip_forward sysctl,
which meant that docker wouldn't change the rule.  But if we're using an
image that preinstalled docker, docker might have already reconfigured
iptables before we run, and we didn't set it back.

We now set it back.

kubernetes/kubernetes#40182
@gtaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 21, 2017

To make this easier for those in the future: this got merged and released in 1.8.4.

magic7s referenced this issue in magic7s/ansible-kubeadm-contiv Feb 12, 2018
GabyCT added a commit to GabyCT/tests-1 that referenced this issue Jul 10, 2018
As this issue was already solved kubernetes/kubernetes#40182,
we do not need to perform the sudo iptables -P FORWARD ACCEPT.

Fixes kata-containers#488

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
GabyCT added a commit to GabyCT/tests-1 that referenced this issue Oct 9, 2018
As this issue was already solved kubernetes/kubernetes#40182,
we do not need to perform the sudo iptables -P FORWARD ACCEPT.

Fixes kata-containers#488

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
@QiuMike

This comment has been minimized.

Copy link

commented Jan 31, 2019

FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

this does not make it work even though the packet forward use docker0, because the packet from other node will not be in ctstate RELATED or ESTABLISHED, the first packet will be NEW, and this packet will be drop, so it will never be in ESTABLISHED state

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.