Cilium with IPSec tunneling fails to start on 3033.2.2

[shosti]

Description

Since updating to 3033.2.2, all of my Cilium pods are in a CrashLoopBackoff state with the following error message:

level=fatal msg="IPSec with tunneling requires support for xfrm state output masks (Linux 4.19 or later)." error="invalid argument" subsys=daemon

After rolling back to 3033.2.1, cilium starts up again.

Impact

Cilium fails -> all other pods can't get network -> general mayhem 😈

Environment and steps to reproduce

1. Set-up:

• Flatcar Linux 3033.2.2 on x86s
• Cilium set up as CNI with the cilium/cilium Helm chart at 1.11.0 and the following Helm values: https://gist.github.com/shosti/f8c93283a200af0f8dd9de0f73f794bd

2. Task: Automatic upgrade
3. Action(s): Update from 3033.2.1 to 3033.2.2, check kubectl logs for a cilium pod
4. Error:

level=fatal msg="IPSec with tunneling requires support for xfrm state output masks (Linux 4.19 or later)." error="invalid argument" subsys=daemon

Expected behavior

Cilium pods start up correctly.

[pothos]

Thanks for the report, maybe our cilium tests need some extensions to catch these cases in the future.
It looks like these xfrm changes could be related: https://lwn.net/Articles/882912/ (Linux 5.10.94)

[pothos]

In the mean time make sure you disable automatic updates: https://www.flatcar.org/docs/latest/setup/releases/update-strategies/#disable-automatic-updates

[pothos]

Looked at the kernel config under /proc and there is no unexpected difference between the two versions (as was the case with a bugfix kernel update some time ago)

[tormath1]

here's a minimal repro: https://gist.github.com/tormath1/bf3af973de9a4232698cc42199496496 from cilium failing part.

with strace, we can see that we're hitting the following change: torvalds/linux@8dce439

Investigating on the netlink side...

[jepio]

@tormath1 if its failing on the changelink then its a kernel bug (if_id is always 0).

[tormath1]

@jepio based on the linked commit, if_id needs to be different from 0... On the repro, it works fine by setting an explicit if_id to 1 for example.

[jepio]

Missed the lower half of the gist on my phone xD

[pothos]

It's really something to discuss how this (LTS) bug fix kernel update ended up in this situation given the leading principle that userspace ABI should not be broken. Maybe this change could be reverted in the next kernel bug fix release?
@borkmann - do you have an opinion on this from the Cilium (and kernel) side?

Edit: also posted to lkml: https://marc.info/?l=linux-kernel&m=164483790014524&w=2

[tormath1]

Hi @shosti, Cilium's PR has been merged - if you're feeling adventurous, you can build a cilium image and try it to deploy by updating this value: https://github.com/cilium/cilium/blob/v1.11.0/install/kubernetes/cilium/values.yaml#L84

Otherwise, the fix should be part of the next cilium's release (1.11.3). :)

[pothos]

The proposal to revert upstream got rejected but we decided to do it for Flatcar anyway: flatcar-archive/coreos-overlay#1682 - this is not part of a release yet.

[tormath1]

Hi @shosti , so it seems that cilium has already fixed the issue starting from latest release 1.11.2. Do you think you could give a try ?
I tested with IPSec test case: flatcar/mantle#292 and it seems to work as expected.

[pothos]

Maybe as additional info that there is a regression: cilium/cilium#19019

Unfortunately, this workaround breaks IPsec connectivity between nodes. Once the XFRMA_IF_ID is set to the placeholder value (1), traffic that should be encrypted leave the node without any encryption. On GKE and self-managed clusters, that's the only noticeable impact. However, on AKS and EKS, we also have BPF logic to rewrite the outer IP address to the proper IP. This still happens despite the failure to encrypt traffic, leading to packet drops.

Edit: With 1.11.2 this is not a problem

[pothos]

I think we can close this now as a Cilium release was done and we have anyway reverted the change for next week's release