-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to reach the kube-dns from external workload #32517
Comments
Thanks for your thorough explanation! However, we generally shy away from such complex topologies, as things can easily go wrong. I don't know that any of us have enough experience with WSL and docker on WSL. At the end of the day, our primary use-case is Kubernetes on Linux directly, and virtualized environments are generally best-effort for development. Do you have any idea where the packets are being dropped? |
Installed tshark to see if any packets are send from the external workload. looks like the DNS packets are send to the coredns pod:
Using Tsharp on the cluster node doenst seem to capture any packets, and my attempts at starting Tshark in the coredns pod hasn't been succesfull. About the complexity of the envourment, I also tried it on a AWS vpc with 1 node and 1 external-workload. both ubuntu, docker and the cluster spun up with minikube. I still expierence the same problem. |
I agree with squeed that this setup is probably out of scope of the issue tracker. Can you try building a simple reproducer and then asking for help on Slack? |
I created a setup with real kubernetes node (not minikube, k3s, kind, etc) in an AWS VPC. added an VM in the same network segment as the node. changed the Tried to change the Loadbalancer service to a NodePort on my minikube setup. it wasnt able to connect. didn't do any further troubleshooting. my conclusion: dont use minikube for testing external workloads. |
Is there an existing issue for this?
What happened?
Environment:
K8-node:
Running minikube on my WSL2 Ubuntu instance with a custom compiled kernel created with this manual https://wsl.dev/wslcilium/, stopped at checkpoint 1 as the rest was irrelevant for my setup. Running this in WSL2 instead of VirtualBox because of performance issues.
External workload:
Running ubuntu VM in VirtualBox with one NAT interface.
OpenVPN:
Used OpenVPN server (deamon) setup on the WSL2 ubuntu instance, using a "tun" interface to directly connect the external VM to the K8-node. See config file below.
https://docs.cilium.io/en/v1.15/network/external-workloads/ is the manual I use to set up the external workload setup.
Spinning up a cluster with these commands:
minikube start --network-plugin=cni --enable-default-cni=false
cilium install --version 1.15.4 --set routingMode=tunnel
cilium clustermesh enable --service-type LoadBalancer --enable-external-workloads
minikube tunnel -c --bind-address 172.16.0.1
# ip adres of the OpenVPN tun interface. shows 127.0.0.1 on external IP clustermesh-apiserver LoadBalancer service, but is reachable trough the VPN.cilium clustermesh vm create external-vm -n default --ipv4-alloc-cidr 192.168.69.0/30
cilium clustermesh vm install install-external-workload.sh
# change "CLUSTER_ADDR" to 172.16.0.1 after creation of file. Copy to external VMCommand on External Workload:
sudo HOST_IP=172.16.0.2 ./install-external-workload.sh
Network:
What I tried
I know the 10.96.00/24 network isnt known in the routing table of the external workload. So I tried 2 things:
Both didnt work. Also changed my network setup so that the theres only one 10.0.0.0/8 ip range in my whole setup (default OpenVPN subnet is "10.8.0.0/24", etc), so the 10.0.0.0/8 route would could work withoud conficting routes.
Output:
K8-node:
External Workload:
OpenVPN server.conf file:
Cilium Version
on minikube node:
cilium-cli: v0.16.4 compiled with go1.22.1 on linux/amd64
cilium image (default): v1.15.3
cilium image (stable): v1.15.4
cilium image (running): 1.15.4
on external workload:
Client: 1.15.4 9b3f9a8 2024-04-11T17:25:42-04:00 go version go1.21.9 linux/amd64
Daemon: 1.15.4 9b3f9a8 2024-04-11T17:25:42-04:00 go version go1.21.9 linux/amd64
Kernel Version
on minikube node:
Linux ubuntu 5.15.153.1-microsoft-standard-WSL2+ #1 SMP Tue Apr 30 09:50:49 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux
on external workload:
Linux external-vm 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux###
Kubernetes Version
on k8-node:
minikube version: v1.33.0
commit: 86fc9d54fca63f295d8737c8eacdbb7987e89c67
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
on external workload:
Docker version 26.1.1, build 4cf5afa
Regression
No response
Sysdump
cilium-sysdump-20240513-154955.zip
Relevant log output
No response
Anything else?
No response
Cilium Users Document
Code of Conduct
The text was updated successfully, but these errors were encountered: