[rootless] svclb-traefik CrashLoopBackOff: /usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system

[AkihiroSuda]

Environmental Info:
K3s Version: v1.19.3+k3s1 (974ad30)

Node(s) CPU architecture, OS, and Version: Ubuntu 20.04 (kernel 5.4.0-51-generic), x86_64

Cluster Configuration: single-node rootless

Describe the bug:

svclb-traefik is causing CrashLoopBackOff

Steps To Reproduce:

$ k3s server --rootless

$ k3s kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS             RESTARTS   AGE
kube-system   local-path-provisioner-7ff9579c6-z6kkc   1/1     Running            0          2m9s
kube-system   metrics-server-7b4f8b595-tgl9r           1/1     Running            0          2m9s
kube-system   coredns-66c464876b-g8f42                 1/1     Running            0          2m9s
kube-system   helm-install-traefik-4g4ct               0/1     Completed          0          2m9s
kube-system   traefik-5dd496474-5lmbp                  1/1     Running            0          102s
kube-system   svclb-traefik-xghpq                      0/2     CrashLoopBackOff   6          102s

$ k3s kubectl  -n kube-system logs  svclb-traefik-xghpq  lb-port-80
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

$ k3s kubectl  -n kube-system logs  svclb-traefik-xghpq  lb-port-443
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

Expected behavior:

Shouldn't cause CrashLoopBackOff

Actual behavior:

CrashLoopBackOff is happening

[AkihiroSuda]

The issue is also reproducible with k3s-1.19.1+k3s1

[AkihiroSuda]

A workaround: sudo sysctl net.ipv4.ip_forward=1

Theoretically k3s should not need this sudo sysctl.
Even if it was required, k3s should fail earlier with a human-readable error message.

[j-landru]

Does this issue similar to the CrashLoopBackOff issues on klipper-lb with Calico CNI ? described on k3s-io/klipper-lb#6 and k3s-io/klipper-lb#4

[dorinclisu]

I am getting the same issue with single node standard fresh install, x86 DigitalOcean droplet, v1.20.4
k3s is already running as root.

I'm not sure if this error is ignorable, considering that I couldn't get the LB / ingress to work (returns 500 server error) but I cannot rule out anything..

[brandond]

I have heard other users say that they have been unable to get k3s to work in DO because it is a VPS and does not have sufficient access to load kernel modules or alter sysctls.

[dorinclisu]

Interesting, especially since using standard droplets is presented here https://rancher.com/blog/2020/k3s-high-availability
It was using v1.19.3 though at the time of writing.

[dorinclisu]

I don't mind paying for managed k3s, but my requirement is to be able to join my own (on-premise) worker nodes. I am not sure what are the options here.

[dorinclisu]

I joined a bare metal ARM machine (Armbian 20.11.6 Buster) to my DO master to rule out VPS restrictions, and checked the logs on the lb pod (same error for 443 container):

+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.247.127/32 -p TCP --dport 80 -j DNAT --to 10.43.247.127:80
+ iptables -t nat -I POSTROUTING -d 10.43.247.127/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

This is clearly an open issue, should I create a new one or?

[brandond]

If the VPS configuration doesn't allow you to enable IP forwarding or adjust other kernel settings, there's not much we can do about it. These are all prerequisites for running K3s or any Kubernetes distro.

The blog post is using doctl compute droplet create --image ubuntu-20-04-x64 --size s-1vcpu-2gb --region lon1 - how does this compare with the instance size and base image you're using?

[dorinclisu]

The droplet is 2 GB Memory / 25 GB Disk / FRA1 - Ubuntu 20.04 (LTS) x64.

The fact is, running sysctl net.ipv4.ip_forward on both the droplet and the bare metal armbian returns net.ipv4.ip_forward = 1, so it is enabled.

[AkihiroSuda]

Rootful is off-topic in this issue, I suggest opening another issue for rootful errors.

[dorinclisu]

Opened #3116