Flannel CNI fails if ipv6 is disable in kernel

[qrpike]

I disable ipv6 as one of the boot options for the kernel. I have been using flannel for a while now, and it's been working fine. However when I upgrade from 1.8.x to kubernetes 1.9.x it starts failing due to ipv6 sysctl directories not existing.

I'm not sure if this is a flannel or CNI problem, since I'm in the coreos ecosystem I figured I would ask here first.

Expected Behavior

CNI should create the interface in flannel when creating a new pod.

Current Behavior

I have flannel setup with VXLAN with directrouting enabled. When I start the coreos kubelet 1.9.x I get the following:

Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547096    2062 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "node-problem-detector-xqp5k_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547148    2062 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "node-problem-detector-xqp5k_kube-system(7d585c3f-fbee-11e7-b6ee-1eb535d7075a)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "node-problem-detector-xqp5k_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547165    2062 kuberuntime_manager.go:647] createPodSandbox for pod "node-problem-detector-xqp5k_kube-system(7d585c3f-fbee-11e7-b6ee-1eb535d7075a)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "node-problem-detector-xqp5k_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547236    2062 pod_workers.go:186] Error syncing pod 7d585c3f-fbee-11e7-b6ee-1eb535d7075a ("node-problem-detector-xqp5k_kube-system(7d585c3f-fbee-11e7-b6ee-1eb535d7075a)"), skipping: failed to "CreatePodSandbox" for "node-problem-detector-xqp5k_kube-system(7d585c3f-fbee-11e7-b6ee-1eb535d7075a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"node-problem-detector-xqp5k_kube-system(7d585c3f-fbee-11e7-b6ee-1eb535d7075a)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"node-problem-detector-xqp5k_kube-system\" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory"
Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547441    2062 cni.go:259] Error adding network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
Jan 29 22:42:34 blade02 kubelet-wrapper[2062]: E0130 03:42:34.547461    2062 cni.go:227] Error while adding to cni network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory

Possible Solution

Check if the ipv6 directory exists before trying to write to it

Steps to Reproduce (for bugs)

Disable ipv6 as a boot option, try running kubernetes 1.9.x

Context

Your Environment

• Flannel version: v0.10.0
• Backend used (e.g. vxlan or udp):
  VXLAN with directrouting=true
• Etcd version:
  3.2.9
• Kubernetes version (if used):
  1.9.X ( Tried with 1.9.0, 1.9.1, and 1.9.2 coreos images, eg: v1.9.2_coreos.0 )
• Operating System and version:
  Container Linux by CoreOS stable (1576.5.0)
• Link to your project (optional):

[xuanyuan300]

+1

[wangyaliyali]

how to resolve this problem?

[qrpike]

@Wujiandong @wangyaliyali I put in an issue into CNI ( containernetworking/cni#532 )

This is fixed in master, however we are just waiting for a release. see:

• Bridge plugin assumes system has IPv6 enabled containernetworking/plugins#102 (comment)
• Don't touch ipv6 sysctls when there are no v6 addresses containernetworking/plugins#59

[KevinTHU]

it is a kernel problem, reference https://bugzilla.redhat.com/show_bug.cgi?id=1445054, and some comment say

Reproduced on 3.10.0-632.el7 kernel using script in comment 1. Adding ip6_disable=1 to kernel cmdline made vxlan not working.

Verified on 3.10.0-668.el7 kernel. The vxlan works with or without ip6_disable=1 on kernel cmdline.

[PaulFurtado]

@qrpike This fix was merged in containernetworking/plugins#62
And it's in the v0.7.0 release: containernetworking/plugins@v0.6.0...v0.7.0

@KevinTHU That looks like a different bug than the CNI errors shown above

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.