kliper-lb IPv6 compliance issue

[j-landru]

Context
Trying to build a k3os / k3s single node IPv6 only cluster

Describe the issue
As stated in k3s IPv6 issue k3s-io/k3s#284

It seems that svclb-traefik-xxx pod fails to set clean iptables rules for IPv6, if I believe the container logs

# kubectl logs -n kube-system --all-containers pod/svclb-traefik-g4tfl 
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 2001:db8:3111:80:5e45:1ce5:0:42a8/32 -p TCP --dport 80 -j DNAT --to 2001:db8:3111:80:5e45:1ce5:0:42a8:80
iptables v1.6.2: Invalid port:port syntax - use dash

Try `iptables -h' or 'iptables --help' for more information.
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 2001:db8:3111:80:5e45:1ce5:0:42a8/32 -p TCP --dport 443 -j DNAT --to 2001:db8:3111:80:5e45:1ce5:0:42a8:443
iptables v1.6.2: Invalid port:port syntax - use dash

Try `iptables -h' or 'iptables --help' for more information

2001:db8:3111:80:5e45:1ce5:0:42a8:80 and 2001:db8:3111:80:5e45:1ce5:0:42a8:443 are not correct !! Still the same misunderstanding regarding the syntax of IPv6 address when specifying the port in the IPv6 address. Address should be bracket enclosed [2001:db8:3111:80:5e45:1ce5:0:42a8]:443 !! or here maybe --dport explicitly specified; and IPv4 prefix length of "/32" in "-s 2001:db8:3111:80:5e45:1ce5:0:42a8/32" is stangely short for an IPv6 address ?? Shouldn't it be set to "/128" or "/64" ??

klipper-lb doesn't set clean IPv6 iptables rules as IPv6 addresses have to be bracket enclosed when port is specified within the address.

Describe alternatives you've considered
cloning klipper-lb, I built my own klipper-lb image and tried to manage IPv6 address case patching entry script

--- entry.orig  2020-08-13 17:11:23.306338739 +0200
+++ entry       2020-08-18 09:45:20.694567832 +0200
@@ -3,15 +3,36 @@
 
 trap exit TERM INT
 
-echo 1 > /proc/sys/net/ipv4/ip_forward || true
+# 20200818 IPv6 address compliance
+# try to manage IPv6 address case where address has to be square bracket enclosed when specifiying DEST_PORT
+# Nota : bash regex to determine IP address type was found at https://helloacm.com/how-to-valid-ipv6-addresses-using-bash-and-regex/
+#        and translated from bash to sh with example found at https://stackoverflow.com/questions/30647654/how-to-write-and-match-regular-expressions-in-bin-sh-script
+#        alternative could be to use sipcalc and egrep v4 or v6, [ `sipcalc $DEST_IP | egrep v6` ], probably more robust to validate IP address version,
+#        but need to install sipcalc and egrep apk packages in the container image
 
-if [ `cat /proc/sys/net/ipv4/ip_forward` != 1 ]; then
+#if [[ $DEST_IP =~ ^([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$ ]]; then
+if  echo $DEST_IP | grep -Eq '^([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$'; then
+    # echo IPv4 address
+    echo 1 > /proc/sys/net/ipv4/ip_forward || true
+    if [ `cat /proc/sys/net/ipv4/ip_forward` != 1 ]; then
+        exit 1
+    fi
+    iptables -t nat -I PREROUTING ! -s ${DEST_IP}/32 -p ${DEST_PROTO} --dport ${SRC_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT}
+    iptables -t nat -I POSTROUTING -d ${DEST_IP}/32 -p ${DEST_PROTO} -j MASQUERADE
+#elif [[ $DEST_IP =~ ^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$ ]]; then
+elif echo $DEST_IP | grep -Eq '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$'; then
+    # echo IPv6 address
+    echo 1 > /proc/sys/net/ipv6/conf/all/forwarding || true
+    if [ `cat /proc/sys/net/ipv6/conf/all/forwarding` != 1 ]; then
+        exit 1
+    fi
+    iptables -t nat -I PREROUTING ! -s ${DEST_IP}/128 -p ${DEST_PROTO} --dport ${SRC_PORT} -j DNAT --to [${DEST_IP}]:${DEST_PORT}
+    iptables -t nat -I POSTROUTING -d ${DEST_IP}/128 -p ${DEST_PROTO} -j MASQUERADE
+else
+    echo $DEST_IP  " Neither IPv4, nor IPv6 address !!"
     exit 1
 fi
 
-iptables -t nat -I PREROUTING ! -s ${DEST_IP}/32 -p ${DEST_PROTO} --dport ${SRC_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT}
-iptables -t nat -I POSTROUTING -d ${DEST_IP}/32 -p ${DEST_PROTO} -j MASQUERADE
-
 if [ ! -e /pause ]; then
     mkfifo /pause
 fi

The script now detects IPv6 address but exit without setting IPv6 iptables rules as /proc/sys/net/ipv6/conf/all/forwarding is read-only, so svclb-traefik-g4tfl pod still in CrashLoopBockOff state.

kubectl logs -n kube-system pod/svclb-traefik-g4tfl --all-containers
+ trap exit TERM INT
+ echo 2001:db8:3111:80:5e45:1ce5:0:42a8
+ grep -Eq '^([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$'
+ echo 2001:db8:3111:80:5e45:1ce5:0:42a8
+ grep -Eq '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$'
/usr/bin/entry: line 34: can't create /proc/sys/net/ipv6/conf/all/forwarding: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv6/conf/all/forwarding
+ '[' 0 '!=' 1 ]
+ exit 1
+ trap exit TERM INT
+ echo 2001:db8:3111:80:5e45:1ce5:0:42a8
+ grep -Eq '^([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9]{1,2}|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$'
+ echo 2001:db8:3111:80:5e45:1ce5:0:42a8
+ grep -Eq '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$'
/usr/bin/entry: line 34: can't create /proc/sys/net/ipv6/conf/all/forwarding: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv6/conf/all/forwarding
+ '[' 0 '!=' 1 ]
+ exit 1

I don't find why /proc/sys/net/ipv4/ip_forward could be read-write, as original script seems to work in IPv4 context, and /proc/sys/net/ipv6/conf/all/forwarding is read-only. Help needed to investigate further...

[manuelbuil]

This will be fixed by k3s-io/k3s#4114