-
Notifications
You must be signed in to change notification settings - Fork 501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes to kernel module compression can break certain workflows #3968
Comments
(Im adding to this issue to have all informations at one place.) We have a similar issue right now since one of our cluster updated to Bottlerocket v1.20. Can confirm that adding the missing module - in our case it was Deploying a debug pod and resolving any DNS on a node where a coredns pod lives on results in "i/o timeouts" that can be seen in Turns out that an important factor is to have a
Can anyone confirm having DNS issues as well with Bottlerocket v1.20 and cilium? (pods on same host as coredns cant resolve DNS). |
Thanks for opening the issue with Cilium. I've added more details there as well. Was this configuration working for you in Bottlerocket v 1.19.5? There was a change made some time ago which caused DNS issues on binding to Also, Cilium requires the modules listed here to be loaded. Can you validate that they are all loaded? I have updated the issue description with commands to load all the required kernel modules. |
@vigh-m our configuration works with BR 1.19.5.
Only the AMI is different. Just tested to deploy with your changed user_data. Coredns wont start with new configuration complaining about i/o timeouts when trying to read via udp from cluster ip to host ip. Will try with newer cilium version. The required modules where loaded in BR 1.19.5 in 1.20 only Next Up: (saw over at cilium issue tracker someone mentioned they have the same issue with 1.15.3 but will try anyway as our configuration might be different) |
@vigh-m today i investigated a bit further and followed your path first. I think you are on the right track here regarding missing modules or modules that aren't loaded. Since i had no luck using your The relevant requirements for cilium are as followed:
This result is identical for To reproduce:
--- lsmod-1.19_diff.txt 2024-05-21 16:04:32.422565616 +0200
+++ lsmod-1.20_diff.txt 2024-05-21 16:04:39.150647799 +0200
@@ -1,22 +1,3 @@
- rpcsec_gss_krb5 0
- auth_rpcgss 1 rpcsec_gss_krb5
- nfsv4 0
- dns_resolver 1 nfsv4
- nfs 1 nfsv4
- lockd 1 nfs
- grace 1 lockd
- sunrpc 5 rpcsec_gss_krb5,auth_rpcgss,nfsv4,nfs,lockd
- fscache 1 nfs
-nf_conntrack_netlink 0
- nft_chain_nat 0
- nft_counter 0
- xt_addrtype 0
- nft_compat 0
- br_netfilter 0
- bridge 1 br_netfilter
- stp 1 bridge
- llc 2 bridge,stp
- tls 0
xt_TPROXY 2
nf_tproxy_ipv6 1 xt_TPROXY
nf_tproxy_ipv4 1 xt_TPROXY
@@ -24,29 +5,25 @@
xt_nat 2
xt_MASQUERADE 1
xt_CT 5
- xt_mark 19
- cls_bpf 17
- sch_ingress 9
+ xt_mark 17
+ iptable_raw 1
+ cls_bpf 19
+ sch_ingress 10
vxlan 0
ip6_udp_tunnel 1 vxlan
udp_tunnel 1 vxlan
xfrm_user 1
xfrm_algo 1 xfrm_user
veth 0
- xt_socket 1
- nf_socket_ipv4 1 xt_socket
- nf_socket_ipv6 1 xt_socket
- ip6table_raw 0
- iptable_raw 1
- nf_tables 3 nft_chain_nat,nft_counter,nft_compat
- nfnetlink 5 nf_conntrack_netlink,nft_compat,ip_set,nf_tables
+ nf_tables 0
+ nfnetlink 3 ip_set,nf_tables
ip6table_filter 1
ip6table_nat 1
iptable_nat 1
- nf_nat 5 nft_chain_nat,xt_nat,xt_MASQUERADE,ip6table_nat,iptable_nat
+ nf_nat 4 xt_nat,xt_MASQUERADE,ip6table_nat,iptable_nat
ip6table_mangle 1
xt_conntrack 1
- xt_comment 33
+ xt_comment 32
iptable_mangle 1
iptable_filter 1
drm 0
@@ -55,21 +32,21 @@
squashfs 2
lz4_decompress 1 squashfs
loop 4
- overlay 34
+ overlay 29
crc32_pclmul 0
- crc32c_intel 5
+ crc32c_intel 4
ghash_clmulni_intel 0
aesni_intel 0
- ena 0
crypto_simd 1 aesni_intel
+ ena 0
cryptd 2 ghash_clmulni_intel,crypto_simd
ptp 1 ena
pps_core 1 ptp
button 0
- sch_fq_codel 5
- nf_conntrack 6 nf_conntrack_netlink,xt_nat,xt_MASQUERADE,xt_CT,nf_nat,xt_conntrack
- nf_defrag_ipv6 3 xt_TPROXY,xt_socket,nf_conntrack
- nf_defrag_ipv4 3 xt_TPROXY,xt_socket,nf_conntrack
+ sch_fq_codel 3
+ nf_conntrack 5 xt_nat,xt_MASQUERADE,xt_CT,nf_nat,xt_conntrack
+ nf_defrag_ipv6 2 xt_TPROXY,nf_conntrack
+ nf_defrag_ipv4 2 xt_TPROXY,nf_conntrack
fuse 1
configfs 1
dmi_sysfs 0 My guess would be that the following modules have to be loaded as well or network policies will not work. They did get loaded in bottlerocket
As mentioned here https://docs.cilium.io/en/stable/operations/system_requirements/#requirements-for-l7-and-fqdn-policies @vigh-m though we are seeing different modules not loaded i would concur with your initial finding regarding i believe i do not need to update to a later version of cilium right now to furhter strengthen the findings of yours. Seems pretty clear to me that the underlying issue is in fact modules not beeing loaded thus breaking essential features of Regards, |
Hi @obirhuppertz, Thank you so much for your deep dive on this. It seems pretty clear that the kernel modules not being loaded are the root cause of the issue. I don't think different versions of cilium will behave differently here. The list of modules in the initial issue comment is non-exhaustive and is causing the other failures you are seeing. As we discover more such missing modules, that list is bound to get longer. We're investigating options to build a version of |
Upgrading to 1.20 has broken Antrea with
Will investigate adding this module to user-data |
Reopening since most of these changes have been reverted because of an issue found during testing with Alpine Linux, where the injected mount overwrote the |
Based on a survey of container images, it seems like |
Bottlerocket 1.20.1 release contains a new version of kmod built statically to allow customer containers to mount it. This will ensure that the OS provided kernel modules can be loaded by the container without compatibility issues.
This should allow containers to be launched with the correct packages and modules to work with Cilium and any other such workloads. This is a temporary workaround to enable existing workloads. A more permanent fix is in the works. |
@vigh-m Thanks for the Update! Can confirm that adding the following lines to the
|
Thank you @vigh-m. This fix resolves the issue only partially for us because we're still getting 5% of the requests fail rate with the same timeout error. Some smaller bottlerocket-based nodes fail to load the module on Cilium startup:
So the fix would be using userdata to load the required modules reliably. |
Can you share some more details about your setup? Things like, cilium version, how many t3.medium instances are showing this issue, kubernetes version, and anything else you think is relevant. On my end, I ran a test on 1000
|
@vigh-m Here is the instance configuration:
The nodes have these pods running:
And they have a taint as well:
And here is the userdata that we use for these nodes:
|
I've been trying to reproduce the error you see but haven't had much success. This is after using the same Bottlerocket configs you shared. I am using the Cilium Quick Installation method via the
Are you able to log into the
Also the modprobe should be a static executable
|
Bottlerocket release v1.20.2 contains changes to the default files mounted to containers. The change mounts bottlerocket provided, statically linked This should resolve issues with Cilium, Antrea, and similar workloads being unable to modify kernel modules, without requiring any Bottlerocket specific changes to configurations. |
Image I'm using:
All Bottlerocket OS 1.20.0 variants
What I expected to happen:
Updating to Bottlerocket v1.20.0 should not have broken any existing container workflows.
What actually happened:
Launching Bottlerocket v1.20.0 can result in unexpected errors like:
or
Root Cause:
Following changes in Bottlerocket v1.20.0 we dropped support for
xz
compressed modules in favour ofgz
compression. We're seeing failures to launch certain containers which require kernel modules and havemodprobe
built withoutlibz (gzip)
support.Known Affected systems:
Workaround:
A workaround would be to use user data on new launches or
apiclient
in running hosts to setupkernel.modules.<name>.autoload
to load the required modules as described here.In the case of Cilium the commands would like the following when using the apiclient:
OR via
user-data.toml
:The text was updated successfully, but these errors were encountered: