-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Server nodes behind NAT, pod networking is broken #10011
Comments
Did you set --node-ip and --node-external-ip to the correct values for each of the agents, or just the servers? Based on the information you shared, it sounds like the apiserver is trying to connect to the kubelet's external IP to get logs. Normally it would connect to the public IP using the agent tunnel, so I suspect that the internal and external IPs are not being set properly. |
Thanks for the help Brandon! For the servers, node-ip defaults to the private IP (I believed) which would be 192.x.x.. Should I set node-ip and node-external-ip specifically to their public addresses? This is the current setup
|
Related to #7355 I think. Base on the comment #7355 (comment), I'm still unable to get it working right. |
Thanks for linking that issue @tdtgit ! Since I only have one NAT gateway, I only have one public IP address. So this is my server config:
My agent config:
Here is some additional information:
What I can see from this is, is that server three has only peered with one other server instead of two! |
I've indeed managed to fix this by assigning public IPs to all of my servers. I can read the logs from all pods except the ones on |
but some agent node on nat backend, not public IP. |
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions. |
My edge nodes do not have public IP addresses, peer: 6e/mB peer: AaX5oW4pt1X |
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions. |
Environmental Info:
K3s Version:
k3s version v1.29.3+k3s1 (8aecc26)
go version go1.21.8
Node(s) CPU architecture, OS, and Version:
Arm 64
Amd 64
Ubuntu 22.04
Cluster Configuration:
3 servers, software defined networking behind NAT with public IP
3 public agents
flannel backend wireguard-native, ports are allowed in NAT and nodes
servers are tagged with external IP flag which is set to the NATs IP
Describe the bug:
Multiple:
Pods from the agent nodes can't reach pods on the server nodes.
Can't get logs from server nodes due to :
➜ ~ kubectl -n kube-system logs metallb-speaker-gnc5j Error from server: Get "https://<public-ip-nat>:10250/containerLogs/kube-system/metallb-speaker-gnc5j/metallb-speaker": proxy error from 127.0.0.1:6443 while dialing <public-ip-nat>:10250, code 502: 502 Bad Gateway
Steps To Reproduce:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --flannel-backend wireguard-native --token <token> --disable servicelb --write-kubeconfig-mode 644 --node-external-ip <public-ip-nat> --flannel-external-ip --disable traefik
Installed a helm chart for metallb, and a daemonset for speakers is deployed
Expected behavior:
The pods are able to communicate with each other, I'm able to get logs from all pods.
Actual behavior:
As described in "bug", pods can't speak to each other and I can't get logs from pods on the server nodes
Additional context / logs:
Which logs would help? Happy to supply whatever is needed
The text was updated successfully, but these errors were encountered: