CrashLoopBackOff on Calico and Canal

[sauber]

When using Calico og Canal as the cluster network driver, klipper-lb fails to start up pods for LoadBalancer services.

From a default k3s installation on ubuntu 20.04 with only Flannel disabled, and either Calico or Canal added:

:~$ kubectl -n kube-system get ds/svclb-traefik
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
svclb-traefik   1         1         0       1            0           <none>          13m
:~$ kubectl -n kube-system get pod -l app=svclb-traefik
NAME                  READY   STATUS             RESTARTS   AGE
svclb-traefik-85mxm   0/2     CrashLoopBackOff   14         13m
:~$ kubectl -n kube-system logs ds/svclb-traefik lb-port-80
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1
:~$

[BlankZhu]

When using Calico og Canal as the cluster network driver, klipper-lb fails to start up pods for LoadBalancer services.

From a default k3s installation on ubuntu 20.04 with only Flannel disabled, and either Calico or Canal added:

:~$ kubectl -n kube-system get ds/svclb-traefik
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
svclb-traefik   1         1         0       1            0           <none>          13m
:~$ kubectl -n kube-system get pod -l app=svclb-traefik
NAME                  READY   STATUS             RESTARTS   AGE
svclb-traefik-85mxm   0/2     CrashLoopBackOff   14         13m
:~$ kubectl -n kube-system logs ds/svclb-traefik lb-port-80
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1
:~$

Calico uses "allow_ip_forwarding": false as default.
Try adding "allow_ip_forwarding": true to calico's cni_network_config like below:

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins":
    [
      {
        "type": "calico",
        "log_level": "info",
        "datastore_type": "kubernetes",
        "nodename": "__KUBERNETES_NODE_NAME__",
        "mtu": __CNI_MTU__,
        "ipam": { "type": "calico-ipam" },
        "policy": { "type": "k8s" },
        "kubernetes": { "kubeconfig": "__KUBECONFIG_FILEPATH__" },
        "container_settings": { "allow_ip_forwarding": true },  # ADD THIS
      },
      {
        "type": "portmap",
        "snat": true,
        "capabilities": { "portMappings": true },
      },
      { "type": "bandwidth", "capabilities": { "bandwidth": true } },
    ],
}

The container should run normally.

It works for me, and, I'm getting some logs from lb-port-80/lb-port-443 like below:

+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.199.158/32 -p TCP --dport 80 -j DNAT --to 10.43.199.158:80
+ iptables -t nat -I POSTROUTING -d 10.43.199.158/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

Anyway, my Ingress works normally and the status of lb-port-80/lb-port-443 is Running. Hope this helps.

[sauber]

Thank you. This works.

For future reference, in case other people might run into same problem, to test, I followed the instructions on https://docs.projectcalico.org/getting-started/kubernetes/k3s/quickstart, with the following changes:

1. k3s is installed with traefik enabled
2. Installing using manifest. Manifest is downloaded, '"container_settings": { "allow_ip_forwarding": true }' is inserted in manifest, and manifest is applied. I could not find a way to make the operator method work.
3. Log from svclb-traefik says:

~# kubectl -n kube-system logs ds/svclb-traefik lb-port-80
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.226.7/32 -p TCP --dport 80 -j DNAT --to 10.43.226.7:80
+ iptables -t nat -I POSTROUTING -d 10.43.226.7/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

4. Traffic is forwarded through klipper to traefik service:

/ % curl http://192.168.64.64/
404 page not found

[j-landru]

I'm just trying to give it a try, for an IPv6 only k3s cluster, I restart my with a fresh install on k3os 0.22.2 (k3s 1.22.2), i fall again on that svclb-traefik CrashLoopBackOff just after Calico install. ip_forward is setted to 1 on the node, but svclb-traefik always complain about it.

@sauber An alternate way, to set the CNI to propagate ip_forward inside the pod, for the same result :
As you, I followed the instructions on https://docs.projectcalico.org/getting-started/kubernetes/k3s/quickstart. To workaround that issue, I adjusted the calicoNetwork stanza inside custom-resources.yaml adding
containerIPForwarding: Enabled
before applying the manifest.
On k3os the ephemeral /etc/cni/net.d/10-calico.conflist file has now
"container_settings": { "allow_ip_forwarding": true }
and svclb-traefik is now in running state.

[webermax]

2. Installing using manifest. Manifest is downloaded, '"container_settings": { "allow_ip_forwarding": true }' is inserted in manifest, and manifest is applied. I could not find a way to make the operator method work.

Still holds true for the manifest approach (no idea about the operator mode, neither): gitpod proxy does not work without allowing ip forwarding which is disabled by default.

Meanwhile I left traefik disabled.

[ShaunMWallace]

Same issue here. Using the operator, I just had to set containerIPForwarding to Enabled in the custom-resource.yml and recycle the svclb-traefik DaemonSet and everything came right up