Add missing kernel config checks #7118

caroline-suse-rancher · 2023-03-20T20:35:51Z

This is an issue to track the changes made by @st3iny (thank you!!)

copy-pasta -

Add additional kernel config checks for NETFILTER_XT_MATCH_COMMENT and NETFILTER_XT_MATCH_MULTIPORT as they are both required to run k3s.

Proposed Changes
I was experimenting with minimal linux kernels and k3s the other day. I used your excellent check-config.sh script to check my configuration. However, k3s didn't run and required extra config options that were missing from the script.

Types of Changes
I added 2 missing config checks to the script. I categorized them as required as the server failed to start without them. Furthermore, I bootstrapped vanilla servers without any extra configuration (e.g. env variables) during my testing.

Verification
Here are some error messages taken from journalctl -u k3s.service.

Regarding NETFILTER_XT_MATCH_COMMENT:
Feb 11 22:57:15 virtoo k3s[6375]: panic: F0211 22:57:15.908590 6375 network_policy_controller.go:341] Failed to run iptables command to insert in INPUT chain running [/usr/bin/iptables -t filter -I INPUT 1 -m comment --comment kube-router netpol - 4IA2OSFRMVNDXBVV -j KUBE-ROUTER-INPUT --wait]: exit status 1: Warning: Extension comment revision 0 not supported, missing kernel module?

Regarding NETFILTER_XT_MATCH_MULTIPORT:
Feb 11 23:02:34 virtoo k3s[2678]: panic: F0211 23:02:34.075017 2678 network_policy_controller.go:336] Failed to verify rule exists in KUBE-ROUTER-INPUT chain due to running [/usr/bin/iptables -t filter -C KUBE-ROUTER-INPUT -p tcp -m comment --comment allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN --wait]: exit status 2: Warning: Extension multiport revision 0 not supported, missing kernel module?

Those error messages vanish and the server starts when both match extensions are compiled into the kernel (or as modules).

Testing
I don't think this is subject to unit testing. Please let me know otherwise.

Linked Issues
User-Facing Change
NONE
Further Comments

VestigeJ · 2023-03-21T00:08:46Z

Note failed links are artifacts of testing on Rocky 9.1 for both the reproduction and validations.

Attention: new NETFILTER_XT_MATCH_COMMENT and NETFILTER_XT_MATCH_MULTIPORT missing on the previous release.

Previous VERSION=v1.26.2+k3s1 check-config flags

$ kgn

NAME                                          STATUS   ROLES                  AGE   VERSION
ip-UH-HUH-ONE   Ready    control-plane,master   8s    v1.26.2+k3s1

$ k3s -v

k3s version v1.26.2+k3s1 (ea094d1d)
go version go1.19.6

$ k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/d1373f31227cf763459011fa1123224f72798511c86a56d61a9d4e3c0fa8a0c9/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)

System:
- /var/lib/rancher/k3s/data/d1373f31227cf763459011fa1123224f72798511c86a56d61a9d4e3c0fa8a0c9/bin/aux iptables v1.8.8 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.14.0-162.6.1.el9_1.x86_64
info: reading kernel config from /boot/config-5.14.0-162.6.1.el9_1.x86_64 ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 6 (fail)

Attention - new NETFILTER_XT_MATCH_COMMENT and NETFILTER_XT_MATCH_MULTIPORT present on the latest release-candidate.

Validated new flags on VERSION=v1.26.2+k3s1 check-config

$ k3s -v

k3s version v1.26.3-rc1+k3s1 (7c32f88f)
go version go1.19.7

$ k3s

NAME:
   k3s - Kubernetes, but small and simple

USAGE:
   k3s [global options] command [command options] [arguments...]

VERSION:
   v1.26.3-rc1+k3s1 (7c32f88f)

COMMANDS:
   server           Run management server
   agent            Run node agent
   kubectl          Run kubectl
   crictl           Run crictl
   ctr              Run ctr
   check-config     Run config check
   token            Manage bootstrap tokens
   etcd-snapshot    Trigger an immediate etcd snapshot
   secrets-encrypt  Control secrets encryption and keys rotation
   certificate      Manage K3s certificates
   completion       Install shell completion script
   help, h          Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                     (logging) Turn on debug logs [$K3S_DEBUG]
   --data-dir value, -d value  (data) Folder to hold state (default: /var/lib/rancher/k3s or ${HOME}/.rancher/k3s if not root)
   --help, -h                  show help
   --version, -v               print the version

$ k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/116dd41b01ad045ae4aa9121d6e21db698c700e305030fbb1009f2f1d06b5421/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)

System:
- /var/lib/rancher/k3s/data/116dd41b01ad045ae4aa9121d6e21db698c700e305030fbb1009f2f1d06b5421/bin/aux iptables v1.8.8 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.14.0-162.6.1.el9_1.x86_64
info: reading kernel config from /boot/config-5.14.0-162.6.1.el9_1.x86_64 ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 6 (fail)

caroline-suse-rancher added this to K3s Development Mar 20, 2023

caroline-suse-rancher moved this to To Test in K3s Development Mar 20, 2023

caroline-suse-rancher added this to the v1.26.3+k3s1 milestone Mar 20, 2023

This was referenced Mar 20, 2023

[release-1.25] Add missing kernel config checks #7119

Closed

[release-1.24] Add missing kernel config checks #7120

Closed

VestigeJ self-assigned this Mar 20, 2023

VestigeJ closed this as completed Mar 21, 2023

github-project-automation bot moved this from To Test to Done Issue in K3s Development Mar 21, 2023

nekorouter mentioned this issue Apr 26, 2024

LicheePi 4A Console 在使用 lxd init 的时候，提示缺少内核模块（When using the LicheePi 4A Console with lxd init, it prompts for a missing kernel module.） revyos/revyos#63

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing kernel config checks #7118

Add missing kernel config checks #7118

caroline-suse-rancher commented Mar 20, 2023

VestigeJ commented Mar 21, 2023

Previous VERSION=v1.26.2+k3s1 check-config flags

Validated new flags on VERSION=v1.26.2+k3s1 check-config