Error cleaning up old/bad masquerade rules: failed to lookup iptables rule / CNI not working when run-router=false

[XANi]

What happened?

Error on cleaning iptables at start, and it appears that cni in /var/lib/cni/networks/ is also not initialized when that happens

The problem happened on fresh node install

Jan 27 17:21:38 d1-fw-kube19 kube-router[5598]: I0127 17:21:38.449293    5598 version.go:21] Running /usr/local/bin/kube-router version 1.5.3, built on 2022-11-20T17:55:01Z, go1.19.2
Jan 27 17:21:38 d1-fw-kube19 kube-router[5598]: I0127 17:21:38.653411    5598 metrics_controller.go:164] Starting metrics controller
Jan 27 17:21:38 d1-fw-kube19 kube-router[5598]: I0127 17:21:38.672875    5598 network_services_controller.go:347] Starting network services controller
Jan 27 17:21:38 d1-fw-kube19 kube-router[5598]: E0127 17:21:38.684410    5598 network_services_controller.go:352] Error cleaning up old/bad masquerade rules: failed to lookup iptables rule: running [/usr/sbin/iptables -t nat -C POSTROUTING -m ipvs --ipvs --vdir ORIGINAL --vmethod MASQ -m comment --comment  ! -s  ! -d  -j MASQUERADE --wait]: exit status 2: iptables v1.8.7 (nf_tables): host/network `' not found
Jan 27 17:21:38 d1-fw-kube19 kube-router[5598]: Try `iptables -h' or 'iptables --help' for more information.

running with

ExecStart=/usr/local/bin/kube-router \
    --master=https://1.2.3.4:443/ \
    --run-firewall=false \
    --run-service-proxy=true \
    --run-router=false \
    --kubeconfig=/etc/kubernetes/kube-router.conf \
    --metrics-port 9001

on version 1.5.3 (tested with some earlier ones with same effect) (OS is Debian 11)

and switching --run-router=false to true "fixes" the problem and files in /var/lib/cni/networks/ get created.

we run with --run-router=false because (AFAIK) there is no option to tell kube-router to not inject stuff into system routing table (which we don't want, we already manage that with BIRD)

[aauren]

Thanks for reporting this. It looks like this bug has been around for a while, as far as I know there aren't many people that run kube-router in a proxy only mode, so it's nice when people let us know when this stuff is broken.

It looks like there was a place where we weren't checking for the existence of a variable that is only populated when the router option is selected. #1435 should fix this.

Would you be willing to test out that patch and let me know if it resolves the error you were getting?

As far as /var/lib/cni/networks goes, that is all owned by the CNI you're using. Since you aren't running kube-router in --run-router mode it isn't going to take ownership of your CNI and shouldn't really be doing anything with it. This means that kubelet is going to use whatever you've manually set up in your CNI configuration.

[aauren]

Just to clarify, because the last part of my comment might have been ambiguous, if you're not running kube-router with --run-router it is expected that either you or some other component that you've setup will configure your CNI so that it is able to drive your pod's networking.

[XANi]

@aauren Our core runs on OSPF which means we don't really need any of the BGP stuff, and are using BIRD instance to basically feed the routes from machine to OSPF so additional BGP on top of that is redundant. But we still want to configure rest of the CNI stuff.

Our config is basically k8s nodes connected to VLAN with L3 switch running OSPF

Looking at options now I believe --enable-ibgp=false would cover it I think ?

As for the bug I've compiled 8624776 and there is no iptables errors during start

[aauren]

So the way that kube-router works, is that in order for kube-router to function as a pod networking system (i.e. --run-router) it uses BGP to spread awareness of those routes between other kube-router instances that are running on other nodes.

So if you enable --run-router it will expect that it is able to bind to port 179 on the kubernetes primary node IP and that it will use that, at a minimum to talk to other kube-router instances and advertise pod networking routes and adjust the host's routing table. This is the only way that kube-router functions as a pod networking provider.

It does not have to peer with other routers on your network though if you don't want it to. For this, you just wouldn't configure any external peers.

However, if running BGP at all for kubernetes is unacceptable for your use-case than your only option will be to run some other CNI for pod networking purposes.

kube-router can still work as a firewall or a service proxy without enabling BGP.

[XANi]

@aauren Thanks for clarification how it works! After testing --enable-ibgp=false seems to do exactly what we need.

Initially I assumed --run-router is "just" bgp peering part and cni is separate because --enable-cni option exists