Incompatible ipset protocol version (7) will cause problems

[uablrek]

Expected Behavior

Calico can handle different ipset protocol versions. At least 6-7.

Current Behavior

If K8s upgrades to ipset >= v7.0 outgoing connects from pods fail.

Possible Solution

Upgrade the calico image to include ipset v7.x since it's backward compatible.

Steps to Reproduce (for bugs)

1. Make sure K8s is upgraded to use ipset v7.x (this is non-trivial if installed with kubeadm)
2. Use proxy-mode=ipvs (uses ipset)
3. Try an outgoing connect from a pod

The kube-proxy (v1.22.2) image includes ipset v6.38. It must be changed to some v7.x version. How that is done I can't say. I start kube-proxy as a program (no container) so for me it's simply to change the ipset on the node.

Context

While this isn't a problem at the moment, it will be the day K8s upgrades to iset v7.x

Your Environment

• Calico version: calico/cni:v3.19.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): K8s
• Operating System and version: Own. But Ubuntu 20.04.3 LTS uses ipset v7.5 at the moment
• Link to your project (optional):

[uablrek]

I can help with testing if you provide an updated Calico test-image.

[lwr20]

Pinging @caseydavenport

[uablrek]

Stand-alone test with ipset;

# ipset6 -v
ipset v6.38, protocol version: 6
# ipset7 -v
ipset v7.15, protocol version: 7
# ipset6 create foo6 hash:net,iface
# ipset6 list foo6
Name: foo6
Type: hash:net,iface
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 488
References: 0
Number of entries: 0
Members:
# ipset7 list foo6
Name: foo6
Type: hash:net,iface
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 488
References: 0
Number of entries: 0
Members:
# ipset7 create foo7 hash:net,iface
# ipset6 list
ipset v6.38: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
Kernel and userspace incompatible: settype hash:net,iface with revision 8 not supported by userspace.

[dghubble]

it will be [a problem] the day K8s upgrades to iset v7.x

Kubernetes v1.23.0 kube-proxy does use v.7.x now. Calico v3.21.x Pods show the OP's error and will never be Ready. So the time has come.

ipset --version
ipset v7.10, protocol version: 7

[Dr-Shadow]

There is probably other compatibility issues since protocol version 7 seems to be used in my case.

Ubuntu host -- k8s node :

ubuntu@k8s-master-01:~$ sudo ipset version
ipset v7.10, protocol version: 7

Ubuntu host -- calico-node pod

❯ kubectl exec -it -n calico-system calico-node-zq7kg -- bash
Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init)
[root@k8s-master-01 /]# ipset version
ipset v7.1, protocol version: 7
[root@k8s-master-01 /]# ipset list
Name: KUBE-NODE-PORT-TCP
Type: bitmap:port
Revision: 3
Header: range 0-65535
Size in memory: 8264
References: 0
Number of entries: 0
Members:
ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.

[dghubble]

Our distro marked Calico as not supported with Kuberntes v1.23 due to this incompatibility

[Meroje]

On Fedora 35

2021-12-31 17:55:39.777 [INFO][47966] felix/ipsets.go 312: Retrying after an ipsets update failure... family="inet"
2021-12-31 17:55:39.779 [ERROR][47966] felix/ipsets.go 574: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port,net with revision 8 not supported by userspace.\n"
2021-12-31 17:55:39.779 [WARNING][47966] felix/ipsets.go 322: Failed to resync with dataplane error=exit status 1 family="inet"

[uablrek]

@Meroje What linux kernel version do Fedora 35 use?

I am using linux-5.15.2 and only protocol versions 6-7 are supported, not version 8.
Please see; https://github.com/torvalds/linux/blob/8008293888188c3923f5bd8a69370dae25ed14e5/include/uapi/linux/netfilter/ipset/ip_set.h#L16-L18

[Meroje]

It's on 5.15.6 indeed not even the next 5.16 seem to not have that number

edit I believe this is it https://github.com/torvalds/linux/blob/8008293888188c3923f5bd8a69370dae25ed14e5/net/netfilter/ipset/ip_set_hash_ipportnet.c#L31

[lwr20]

Our distro marked Calico as not supported with Kuberntes v1.23 due to this incompatibility

FWIW, our overnight runs (on ubuntu 20.04) show Calico working just fine with k8s 1.23, so this isn't a blanket incompatibility.

[Dr-Shadow]

Our distro marked Calico as not supported with Kuberntes v1.23 due to this incompatibility

FWIW, our overnight runs (on ubuntu 20.04) show Calico working just fine with k8s 1.23, so this isn't a blanket incompatibility.

That's probably because you're using proxy-mode=iptables

[Meroje]

I got errors when using ipvs mode for dualstack use, iptables is fine on my 1.23 cluster

[Dr-Shadow]

Since CentOS 8 is on EOL right now, this is most likely to be fixed once calico node switches from CentOS 8 to CentOS Stream 9 since it includes ipset v7.11 & kernel 5.14.
CentOS Stream 9 got released a few weeks ago.
EDIT : Minimal version with support for ipset bucketsize in all hash types is v7.7

[dghubble]

Yeah, reverting kube-proxy to iptables mode avoids the issue of course, but we'd like to stick to ipvs. We're using other CNIs until Calico supports this again.

[Dr-Shadow]

I can confirm from Slack that there was some work from @song-jiang for switching the base images to CentOS Stream 9 which would provide a sufficient version of ipset. So at least we can tell that this would be fixed in a future release.

Until then ipvs proxy mode SHOULD be disabled to make Calico work on kubernetes >= 1.23.

[invidian]

Hmm, I wonder what I do wrong that Calico v3.21.2 works fine for me with kube-proxy v1.23.1 in ipvs mode. Could it be that I miss some configuration, so kube-proxy is runing in ipvs mode while calico keeps using iptables?

[Dr-Shadow]

Hmm, I wonder what I do wrong that Calico v3.21.2 works fine for me with kube-proxy v1.23.1 in ipvs mode. Could it be that I miss some configuration, so kube-proxy is runing in ipvs mode while calico keeps using iptables?

Can you check calico-node logs and ipset version on host VS on calico-node pod ?

[uablrek]

Please note that incoming traffic works with proxy-mode=ipvs. Outgoing doesn't.

[invidian]

Can you check calico-node logs and ipset version on host VS on calico-node pod ?

Sure, thanks! Here it is:

$ kex calico-node-fzgvv -c calico-node -- ipset --version
ipset v7.1, protocol version: 7
$ ssh ... sudo ipset --version
ipset v7.6, protocol version: 7
$ kex kube-proxy-99mb7 -- ipset --version
ipset v7.10, protocol version: 7
$ klo calico-node-zs9jh | grep ipvs
2022-01-12 08:53:13.568 [INFO][71] felix/driver.go 78: Kube-proxy in ipvs mode, enabling felix kube-proxy ipvs support.
2022-01-12 08:53:13.892 [INFO][71] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=6 ifaceName="kube-ipvs0" state="up"
2022-01-12 08:53:13.896 [INFO][71] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"kube-ipvs0", State:"up", Index:6}
$ klo kube-proxy-78q2f | grep -i ipvs
I1217 00:03:42.452605       1 server_others.go:269] "Using ipvs Proxier"
I1217 00:03:42.452630       1 server_others.go:271] "Creating dualStackProxier for ipvs"

[Dr-Shadow]

What about the kernel version ?

[invidian]

I'm running 5.10.84-flatcar.

EDIT: Let me know if this is off-topic, maybe would be better to discuss it on Slack or something.

[Dr-Shadow]

Interesting, this kernel version doesn't include bucketsize in all hash types for the ipset kernel module that's why you seem to have no issues.

EDIT : Kernel >= 5.11 includes bucketsize in all hash types.

ipvs proxy mode SHOULD be disabled to make Calico work on kubernetes >= 1.23 AND host kernel >= 5.11

[song-jiang]

FYI: The ipset changes is not part of v3.22.0 since it takes more time to get everything fully tested. It should be in v3.22.1.

[lazystone]

@xmudrii I use metallb, so it's ipvs

[caseydavenport]

Yep that's right, the fix has been merged to master but needs cherry-picking to the v3.22 branch still. Once that's done it will be in v3.22.1.

We didn't get this into v3.22.0 because the fix came in a bit later and we had already entered code freeze.

[leoweiyu]

I am running 1.23.4 with Calico3.22 on Ubuntu 20.04.4 LTS in IPVS mode, and I can confirm this is still NOT fixed.

[lazystone]

I mentioned above that it works for me in IPVS mode. But I was wrong - It starts but outgoing connections fail.

[Dr-Shadow]

I can confirm this would be fixed in the next release.
calico/node:latest (not v3.22.0) is working with ipvs mode once you include it with a custom ImageSet
Have a look here : https://projectcalico.docs.tigera.io/maintenance/image-options/imageset
Or wait for the next release (v3.22.1)

[leoweiyu]

I have just rolled out 3.22.1 and I am still seeing the same error message

kk get daemonset -n kube-system calico-node -o yaml|grep image: image: docker.io/calico/node:v3.22.1 image: docker.io/calico/cni:v3.22.1 image: docker.io/calico/cni:v3.22.1 - image: docker.io/calico/pod2daemon-flexvol:v3.22.1

and still seeing
2022-03-06 22:51:02.647 [ERROR][72103] felix/ipsets.go 574: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port,ip with revision 6 not supported by userspace.\n"

[lwr20]

@leoweiyu Would you mind raising as a new Issue (and link to this one) please?

[leoweiyu]

@lwr20 , sorry was busy for the past few days, yeah the last comment in #5717 make sense to me, i am running k8s on arm system and seems 3.22.1-arm did not get the latest fix as 3.22.1-amd