improve kubeadm preflight message for bridge-nf-call-iptables

[fabriziopandini]

Is this a BUG REPORT or FEATURE REQUEST?

A request for improvement of documentation/UX experience

Versions

kubeadm version (use kubeadm version):
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.5", GitCommit:"490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3", GitTreeState:"clean", BuildDate:"2017-06-14T20:03:38Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.5", GitCommit:"490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3", GitTreeState:"clean", BuildDate:"2017-06-14T20:15:53Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Cloud provider or hardware configuration:
  VirtualBox VM with 2GB RAM

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux localhost.localdomain 3.10.0-514.21.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:
  Environment created using vagrant/VirtualBox and with following Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"

  config.vbguest.no_install = true
  config.vm.box_check_update = false
  config.vm.provider "virtualbox" do |vb|
     # Display the VirtualBox GUI when booting the machine
     vb.gui = false
     vb.memory = "2048"
     vb.cpus = "2"
     vb.linked_clone = true if Vagrant::VERSION =~ /^1.8/
  end
end

What happened?

kubeadm init preflight fails with following output:

[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.6
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Some fatal errors occurred:
	/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`

What you expected to happen?

kubeadm init preflight to provide a more "user friendly" message that helps users to identify fix the problem; additionally, it could be useful to move some point from https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#limitations to https://kubernetes.io/docs/setup/independent/install-kubeadm/

How to reproduce it (as minimally and precisely as possible)?

Use Vagrant/VirtualBox and the vagrant file provided above

Anything else we need to know?

link to lprefligh check in code base
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/preflight/checks.go#L502

problem detected investigating #308

[mrichins]

I had this issue as well, if you want a quick fix, provision your boxes with the following command:

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Basically you need to set your ip tables in CentOS (and other redhat family OSs I assume?)

[gtirloni]

This is documented here: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#limitations

Having kubeadm deal with these OS specifics would certainly be interesting but might increase the scope of the tool too much right now.

[fabriziopandini]

@gtirloni. Many thanks for pointing this out.
Changed the Issue in order from bug to request to improve the UX experience/improve the documentation

[luxas]

@fabriziopandini Feel free to improve the preflight checks

[fabriziopandini]

Now the documentation describes this case as well.

[luxas]

Thanks!

[amit-handa]

I was installing kubeadm on virtualbox vm (centos7). (install instructions: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ )
I was unable to do it. landed here. a short fix was mentioned in comments above

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

unfortunately, it didnt work in my case. It gave 'no such file or directory'.
correct fix is mentioned here
Thanks,

[gtirloni]

@amit-handa which Linux distribution are you running?

EDIT: Never mind, just noticed it's CentOS 7. I spin up a Vagrant box here and can reproduce that.

[paolopal]

I was installing kubeadm on virtualbox vm (centos7). (install instructions: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ )
I was unable to do it. landed here. a short fix was mentioned in comments above

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

unfortunately, it didnt work in my case. It gave 'no such file or directory'.
correct fix is mentioned here
Thanks,

To make it persistent it is better to change the sysctl configuration.
For example in Centos 7 you have to change /usr/lib/sysctl.d/00-system.conf file according to:

# Kernel sysctl configuration file
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Enable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1

[myhro]

unfortunately, it didnt work in my case. It gave 'no such file or directory'.
correct fix is mentioned here
Thanks,

In case the link isn't available in the future, it was a matter of enabling the bridge-netfilter with:

modprobe br_netfilter

[subtubes-io]

In my case:
OS=CentOS_7
kubeadm=1.19.0

What fixed this issue was setting the contents of /usr/lib/sysctl.d/00-system.conf to:

# Kernel sysctl configuration file
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_forward                 = 1

run: sysctl --system

BINGO!
DISCO!

A better solution is to override the values in the /etc/sysctl.d/00-system.conf file vs updating /usr/lib/sysctl.d/00-system.conf directly.

[krisnova]

Adding some context here for our archlinux friends 😸 and to help with search

if you see this error with kubeadm init or in my case while installing without kube-proxy for cilium

kubeadm init --skip-phases=addon/kube-proxy

and you are hitting the following netfilter bridge error:

        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

i was able to check in the br_netfilter kernel module was loaded on my system (provided by iptables-nft in arch) using

lsmod | grep br_netfilter

br_netfilter           32768  0

and then followed the suggestion above to mutate the sysctl config

emacs /usr/lib/sysctl.d/10-arch.conf

and pasted

# Disable netfilter on bridges.                                                 
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_forward                 = 1

and there you go 😄

[renukamanavalan]

Curious!
What would be the case, if I skip "net.bridge.bridge-nf-call-ip6tables" and ignore it explicitly as "--ignore-preflight-errors=FileContent--proc-sys-net-bridge-bridge-nf-call-iptables" in kubeadm init