Mac users unable to keep calico-kube-controller pods up after adding additional application pods

[izorster]

Mac users with local kubernetes single-node cluster installed by k3d using the default calico.yaml:
When fully installing our application chart, are unable to get the calico-kube-controller pods to come up past ContainerCreating once they restart. Complains of "no space left on device" for tmpfs mount in /var/lib/kubelete.pods.... similar to this:

Warning FailedMount 50s kubelet (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-q6jh6" : write /var/lib/kubelet/pods/6ad5f79e-4f7a- │ │ 43ce-a01d-1770fbadd14d/volumes/kubernetes.io\~projected/kube-api-access-q6jh6/..2022_03_09_15_56_14.518548044/token: no space left on device

This works on a linux desktop...

Expected Behavior

There should be no difference between desktop platforms and the pod should work. Unsure why calico-kube-controller pod restarts in the first place.

Current Behavior

See the general description above.

Possible Solution

Unknown...the tmpfs mount seems to default to 64k and I do not know how to increase it.

Steps to Reproduce (for bugs)

 k3d cluster create "$CLUSTER_NAME" \
      --api-port 127.0.0.1:6443 \
      --kubeconfig-update-default=false \
      -p 80:80@loadbalancer \
      -p 443:443@loadbalancer \
      --servers "$NUM_SERVERS" \
      --agents "$NUM_AGENTS" \
      --k3s-arg "--disable=traefik@servers:*" \
      --k3s-arg "--flannel-backend=none@server:*" \
      --k3d_options "--volume" "$(pwd)/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"

4. helm repo add
5. helm repo update
6. helm upgrade --install -n "namespace" "releasename" "chartlocation" --wait --timeout 10m

Context

local testing of application in kubernetes clusters on various desktop platforms (Linux, Mac, VM on Windows)

Your Environment

• Calico version. https://k3d.io/v5.3.0/usage/advanced/calico.yaml
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes 1.25
• Operating System and version: MacOS 12 (Big Sur)
• Link to your project (optional):

[caseydavenport]

This doesn't sound like a Calico issue - you're running your cluster on a local VM and it sounds like that VM doesn't have enough disk space to run all of the pods that you want, and it's causing Calico to fail. You'll need to give your docker VM more disk space.

[izorster]

It's a kubernetes cluster with pods created by the calico.yaml from the website on a MacBookPro. It clearly (based on the df in the info provided) has 70GB and I've allocated up to 12GB of memory. Seems that it is only the calico-kube-controller pod/containers and doesn't happen on a Linux desktop. Feels more like a kubelet/calico issue to me.

[caseydavenport]

It looks really unlikely to be a Calico issue - the problem is occurring before the Calico pod even launches and before the Calico code is even excuted.

I don't see any df output in the original post, but you might want to double check how big your Docker for Mac VM is: https://docs.docker.com/desktop/mac/space/

[ashersilver]

I have the same problem - enough disk, memory and cpus ... can't get calico to stay up.