kernel: enable netfilter ipset (CONFIG_IP_SET + xt_set) — k3s NetworkPolicy enforcement is silently disabled without it

[kingsleydon]

Problem

Running multi-tenant Sysbox workloads under k3s on dstack CVMs (same setup as Dstack-TEE/meta-dstack#80). k3s ships its embedded kube-router NetworkPolicy controller to enforce Kubernetes NetworkPolicy, which requires the netfilter ipset kernel support. The 6.9.0-dstack kernel is built without it, so on startup the controller fails and skips policy enforcement entirely:

ipset save failed: ipset v7.16: Kernel error received: Invalid argument

After this, iptables contains zero KUBE-NWPLCY chains and any applied NetworkPolicy objects are silently unenforced — meaning tenant pods on the same CVM have unrestricted east-west network access to each other. For multi-tenant isolation this is the network-layer counterpart of the disk-quota gap fixed in Dstack-TEE/meta-dstack#80 / #779.

Reproduction: on a sysbox-image CVM running k3s v1.35.x with disable-network-policy: false, apply any NetworkPolicy, then check journalctl -u k3s | grep ipset and iptables-save | grep KUBE-NWPLCY (empty).

Requested change

Add to os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-sysbox.cfg (or dstack.cfg, matching how #779 placed generic facilities), built-in like the quota/loop precedent since the guest has no module-loading tooling:

# Netfilter ipset, required by k3s/kube-router NetworkPolicy enforcement
# for multi-tenant network isolation.
CONFIG_IP_SET=y
CONFIG_IP_SET_HASH_IP=y
CONFIG_IP_SET_HASH_NET=y
CONFIG_IP_SET_HASH_IPPORT=y
CONFIG_IP_SET_HASH_IPPORTIP=y
CONFIG_IP_SET_HASH_IPPORTNET=y
CONFIG_IP_SET_HASH_NETPORT=y
CONFIG_IP_SET_BITMAP_PORT=y
CONFIG_NETFILTER_XT_SET=y

These cover the set types kube-router creates (hash:ip, hash:net, hash:ip,port, ...) plus the -m set iptables match (xt_set).

Alternative considered

An eBPF CNI (Cilium) would avoid ipset, and dstack.cfg already enables CONFIG_BPF/BPF_SYSCALL/BPF_JIT/DEBUG_INFO_BTF — but swapping the CNI is a much larger operational change for guests than enabling the standard netfilter feature the stock k3s path expects. ipset support also benefits any guest using plain iptables+ipset firewalling.

[kvinwang]

Thanks for the detailed report! Good news: most of what you listed is already enabled in the current kernel config (dstack-docker.cfg) — CONFIG_IP_SET, CONFIG_NETFILTER_XT_SET, and the hash:ip / hash:net / hash:ip,port / hash:ip,port,ip / hash:ip,port,net / bitmap set types are all built as modules, and the corresponding kernel module packages (including xt_set) are installed into the rootfs image, where they get auto-loaded on demand — so no in-guest module tooling is required.

The one missing piece was CONFIG_IP_SET_HASH_NETPORT (hash:net,port), which kube-router also creates at startup. A single unsupported set type is enough to make ipset save fail with Kernel error received: Invalid argument and kube-router to bail out of NetworkPolicy enforcement entirely — which matches the behavior you observed.

Opened #793 to add the missing set type (as =m + rootfs module, matching the existing ipset config style rather than built-in). Once that lands in the next guest image, the stock k3s/kube-router NetworkPolicy path should work.