Why CILIUM is still using NOTRACK Kernal Module

[bhunath]

I am trying to Install CILIUM on windows with WSL2.

When I am installing with L7 Proxy I am getting below exception.

level=error msg="Command execution failed" cmd="[iptables -w 5 -t raw -A CILIUM_PRE_raw -m mark --mark 0x00000200/0x00000f00 -m comment --comment cilium: NOTRACK for proxy traffic -j NOTRACK]" error="exit status 2" subsys=iptables
level=warning msg="**iptables v1.8.4 (legacy): Couldn't load target NOTRACK':No such file or directory**" subsys=iptables level=warning subsys=iptables level=warning msg="Try iptables -h' or 'iptables --help' for more information." subsys=iptables
level=error msg="Error while initializing daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon

I check CILIUM IPTABLE Rules File is is giving this Error.
https://github.com/cilium/cilium/blob/master/pkg/datapath/iptables/iptables.go#L605

Linux WSL2 Configuration says they no longer support NOTRACK.
https://github.com/microsoft/WSL2-Linux-Kernel/blob/linux-msft-wsl-4.19.y/net/netfilter/Kconfig#L973

Not able to get around this workflow without disabling L7 Proxy. Which is impacting the visibilty of my PODS.

[pchaigno]

The NOTRACK netfilter target has been deprecated a long time ago (although never removed). However, iptables now transparently converts this target into the CT --notrack target (the replacement). That second target requires CONFIG_NETFILTER_XT_TARGET_CT (corresponding module is xt_conntrack) which is not deprecated. So Cilium doesn't actually require CONFIG_NETFILTER_XT_TARGET_CT.

Maybe WSL2 is missing that second config or maybe the module is not loaded?

[pchaigno]

With #17751, we'll be using -j CT --notrack instead of -j NOTRACK. There should be no functional change since that's what iptables already translated to.

[zaoying]

running cilium 1.11.0 on wsl2(kernel version 5.10.60.1-microsoft-standard-WSL2) still having trouble @pchaigno

level=error msg="Command execution failed" cmd="[iptables -w 5 -t raw -A CILIUM_PRE_raw -m mark --mark 0x00000200/0x00000f00 -m comment --comment cilium: NOTRACK for proxy traffic -j CT --notrack]" error="exit status 2" subsys=iptables
level=warning msg="iptables v1.8.4 (legacy): unknown option \"--notrack\"" subsys=iptables
level=warning msg="Try `iptables -h' or 'iptables --help' for more information." subsys=iptables
level=fatal msg="Error while creating daemon" error="error while initializing daemon: cannot add static proxy rules: exit status 2" subsys=daemon
level=info msg="regenerating all endpoints" reason="kube-apiserver identity updated" subsys=endpoint-manager

[pchaigno]

Is CONFIG_NETFILTER_XT_TARGET_CT enabled? Is the xt_conntrack module loaded?

[hhoover]

Ok I got Cilium and KinD working on WSL2, but I had to compile my own WSL kernel to do it because there are some things in there that as @pchaigno said aren't enabled. This is for the latest at the time of this message, 5.10.60.1. The steps at a high level are:

1. Run a docker container to build your stuff
2. Copy built kernel out of the container to somewhere
3. tell WSL2 to use that kernel

Run a Docker Container to build your stuff

Open up a PowerShell terminal:

docker run --name wsl-kernel-builder --rm -it ubuntu@sha256:9d6a8699fb5c9c39cf08a0871bd6219f0400981c570894cd8cbea30d3424a31f bash

from inside the container (now bash):

WSL_COMMIT_REF=linux-msft-wsl-5.10.60.1
apt update && apt install -y git build-essential flex bison libssl-dev libelf-dev bc

mkdir src
cd src
git init
git remote add origin https://github.com/microsoft/WSL2-Linux-Kernel.git
git config --local gc.auto 0
git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +${WSL_COMMIT_REF}:refs/remotes/origin/build/linux-msft-wsl-5.10.y
git checkout --progress --force -B build/linux-msft-wsl-5.10.y refs/remotes/origin/build/linux-msft-wsl-5.10.y

# adds support for clientIP-based session affinity
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_RECENT is not set/CONFIG_NETFILTER_XT_MATCH_RECENT=y/' Microsoft/config-wsl

# required modules for Cilium
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_CT is not set/CONFIG_NETFILTER_XT_TARGET_CT=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_TPROXY is not set/CONFIG_NETFILTER_XT_TARGET_TPROXY=y/' Microsoft/config-wsl

# build the kernel
make -j2 KCONFIG_CONFIG=Microsoft/config-wsl

Copy that kernel image out of there

Open up a new PowerShell terminal:

docker cp wsl-kernel-builder:/src/arch/x86/boot/bzImage .

This should put it in your %UserProfile% directory (C:\Users\something). You can exit out of the first container now. We're done with that.

Tell WSL2 to use your new kernel

Edit C:\Users\something\.wslconfig

[wsl2]
kernel=C:\\Users\\<your_user>\\bzImage

keep the double slashes. Shut down WSL (in PowerShell: wsl --shutdown) and restart Docker Desktop. From there start up KinD and install Cilium like the install instructions say. DONE

Credit goes to the TKG Community Edition team for easy-to-follow WSL kernel instructions. Those are here and obviously I cribbed it all for this.

[arukiidou]

I could compile the latest WSL kernel (5.10.102.1), from ubuntu22.04 builder, and it was available in WSL2-kind.
Thanks that @hhoover

Build-instructions(Official)

I had installed packages according to build-instructions.
https://github.com/microsoft/WSL2-Linux-Kernel#build-instructions

But, it is missing some packages.

apt-get update && apt-get install -y build-essential flex bison dwarves libssl-dev libelf-dev
make -j2 KCONFIG_CONFIG=Microsoft/config-wsl
#[1] add python-is-python3
/usr/bin/env: 'python3': No such file or directory
#[2] add bc
/bin/sh: 1: bc: not found

Build-instructions(After all)

We can compile the package by installing it

apt-get update && apt-get install -y build-essential flex bison dwarves libssl-dev libelf-dev python-is-python3 bc
make -j 7 KCONFIG_CONFIG=Microsoft/config-wsl

note: I am removing the git package, because I am using gitlab-ci.

Sample: .gitlab-ci.yml

build:
  needs: []
  image: ubuntu:22.04
  before_script:
    - apt-get update && apt-get install -y build-essential flex bison dwarves libssl-dev libelf-dev python-is-python3 bc
  script:
    - make -j 7 KCONFIG_CONFIG=Microsoft/config-wsl
  artifacts:
    paths:
      - arch/x86/boot/bzImage*

Of course, Microsoft/config-wsl need to modified

# adds support for clientIP-based session affinity
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_RECENT is not set/CONFIG_NETFILTER_XT_MATCH_RECENT=y/' Microsoft/config-wsl

# required modules for Cilium
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_CT is not set/CONFIG_NETFILTER_XT_TARGET_CT=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_TPROXY is not set/CONFIG_NETFILTER_XT_TARGET_TPROXY=y/' Microsoft/config-wsl