[HELP] Running Cilium in k3d (bpf-mount) #479
Replies: 18 comments 4 replies
-
|
Hi, thanks for opening this issue 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@iwilltry42 Thanks for looking into this. The install should work without mounting (for non-production use-case), it will get auto-mounted when cilium pod is started. I am also on a mac, don't know if that could work (I also tried just in case and the error is the same). The cilium install works out of the box with |
Beta Was this translation helpful? Give feedback.
-
|
I diffed |
Beta Was this translation helpful? Give feedback.
-
Nice finding! UPDATE: I linked a Google Drive folder with test release files in the PR and will drop it here as well: https://drive.google.com/drive/u/0/folders/1dAvLKlqs5hgXmUnrVs2pn0VzzaMsdKus |
Beta Was this translation helpful? Give feedback.
-
|
Sorry @iwilltry42, this is not it, might be needed but does not solve the current issue. Started investigating now, might be the issue with the base image (k3s), will share the results. |
Beta Was this translation helpful? Give feedback.
-
|
@arjantop-cai , damn.. but it was worth a try 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
Shouldn't it be: Or also |
Beta Was this translation helpful? Give feedback.
-
|
@blaggacao I don't think its that simple, I am running on a mac, there is no If I run the command it (as expected) does not work, but now already fails during k3d startup, not when installing cilium: |
Beta Was this translation helpful? Give feedback.
-
|
Oh, sorry, I overlooked you where working on a Mac. In sone time from now I might be conducting experiments with cillium as well on linux. So I might come back with more insight. |
Beta Was this translation helpful? Give feedback.
-
|
After doing some quick research, I'm afraid we won't be able to resolve this properly on k3d side. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I've been able to fix the k3d issue like this: Then, it does not fail anymore with the problem of accessing the All the best, |
Beta Was this translation helpful? Give feedback.
-
|
well |
Beta Was this translation helpful? Give feedback.
-
|
Hey @iwilltry42 @arjantop-cai @blaggacao , I figured out how you can run Cilium with k3s / k3d on docker for mac. See https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html for the full explanation :-) Feel free to re-use the content for your documentation or so :-) All the best, |
Beta Was this translation helpful? Give feedback.
-
|
@skurfuerst You mention in your linked article that you had to completely restart the "docker-for-mac VM" completely. How do you do that on a Mac? Also, have you made any improvements to your use of Cilium on k3d since this post? Have you tried it with the newest k3d version? |
Beta Was this translation helpful? Give feedback.
-
|
@skurfuerst So the only reason why it is not working is that image does not have bash installed? Should k3d base image have it installed for compatibility? |
Beta Was this translation helpful? Give feedback.
-
|
@arjantop-cai IMHO that would help, but AFAIK this is the IMHO it would actually be better to rewrite the Script on the Cilium side to only use /bin/sh. The script is quite simple, and AFAICS no bash features are used there. All the best, |
Beta Was this translation helpful? Give feedback.
-
|
Hey @skurfuerst great work! Thanks for sharing your insights and the content! |
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth I seem to have resolved this on MacOS with the following:
# file: config.yaml
---
# k3d specific section
# see: https://k3d.io/v5.4.6/usage/configfile/
apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
name: mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`)
servers: 3 # same as `--servers 3`
agents: 0 # same as `--agents 0`
volumes:
- volume: /sys/fs/bpf:/sys/fs/bpf:shared # required for cilium, we need eBPF
nodeFilters:
- server:*
- agent:*
options:
k3s:
extraArgs:
- arg: --disable=traefik,servicelb,network-policy,kube-proxy --no-flannel --flannel-backend=none
nodeFilters:
- server:*
- arg: --write-kubeconfig-mode="0644"
nodeFilters:
- server:*
k3d: # k3d runtime settings
wait: true # wait for cluster to be usable before returining; same as `--wait` (default: true)
timeout: "300s" # wait timeout before aborting; same as `--timeout 300s` which is 5mins
disableLoadbalancer: true # same as `--no-lb`
kubeconfig:
updateDefaultKubeconfig: true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true)
switchCurrentContext: true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true)In each container
for n in $(docker ps | grep 'k3s' | awk '{print $1}'); do \
docker exec -it $n mount bpffs /sys/fs/bpf -t bpf; \
docker exec -it $n mount --make-shared /sys/fs/bpf; \
docker exec -it $n mkdir -p /run/cilium/cgroupv2; \
docker exec -it $n mount -t cgroup2 none /run/cilium/cgroupv2; \
docker exec -it $n mount --make-shared /run/cilium/cgroupv2; \
done;Installing CiliumFor the installation of Cilium inside the cluster I chose to copy across the binary to one of the servers and then run the commands from within. Helm didn't seem to work (got API permission errors after installation). export server=$(docker ps | grep -ie 'k3s.*\sserver' | head -n 1 | awk '{print $1}')
docker cp cilium ${server}:/bin/
docker exec -it $server shWithin the server now export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
cilium install --encryption wireguardOutput of /¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: disabled
\__/¯¯\__/ ClusterMesh: disabled
\__/
DaemonSet cilium
Deployment cilium-operator
Containers: cilium
cilium-operator
Cluster Pods: 0/0 managed by Cilium |
Beta Was this translation helpful? Give feedback.
-
|
@nunix also created a cool blog post on getting Cilium to work in a k3d cluster: https://wsl.dev/wslcilium/#bonus-1-the-k3d-fleet-gets-cilium |
Beta Was this translation helpful? Give feedback.
-
|
Any chance that a solution can be supported by the k3d cli? It would be so nice not to have to do the mounts every time we spin up local clusters that use Cilium :-) |
Beta Was this translation helpful? Give feedback.
-
|
Env Line 49 in fcb3b06 |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, Thanks - will be testing that out! |
Beta Was this translation helpful? Give feedback.
-
What did you do
How was the cluster created?
k3d cluster create test --k3s-server-arg="--flannel-backend=none"What did you do afterwards?
From here: https://docs.cilium.io/en/v1.8/gettingstarted/kind/
Or from here: https://docs.cilium.io/en/v1.8/gettingstarted/k3s/
Error:
kubectl describe pod -nkube-system cilium-5hq8gWhat did you expect to happen
Cilium install should become healthy (it works in k3s: https://docs.cilium.io/en/v1.8/gettingstarted/k3s/)
Which OS & Architecture
Which version of
k3dWhich version of docker
Beta Was this translation helpful? Give feedback.
All reactions