-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GET nodes.metrics.k8s.io fails #1968
Comments
I have the same problem with a fresh install of k3s version v1.17.4+k3s1 running on k3OS v0.10.0 (also on Raspberry Pi 4, with server argument I tried the modifications of This is my yaml file now (maybe I added something in the wrong place?):
This also results in the kubernetes-dashboard pod not starting. |
I don't understand why k3s is contacting random internet servers by default. When apps phone home it tends to make people unpleasant to talk to. If you're lucky they just uninstall it and move on. I just got here and "leave?" is already on my TODO list. |
What do you mean? |
@jdmarshall It sounds like you're under the impression that In this case, 10.43.x.x is used for Kubernetes services running within your cluster, while 10.42.x.x is used for Kubernetes pods. None of this is k3s specific; it's core to how Kubernetes works. If you're having an issue with your k3s cluster, please open a new issue - but perhaps try to avoid jumping to any conclusions about what k3s is or is not doing. |
I'm not seeing IP addresses. I'm seeing repeated errors trying to connect to FQDNs, like v1beta1.metrics.k8s.io Why use a subdomain of a registered internet domain for RFC1918 traffic? That doesn't telegraph 'local address lookup', let alone local/vlan traffic. |
@jdmarshall are you talking about errors like: That's the APIService name. They're all namespaced as part of the Kubernetes standard; it's not a hostname any more than a Java class name with java.sun.com in it is a hostname. See: https://github.com/kubernetes-sigs/metrics-server/blob/master/manifests/base/apiservice.yaml#L5 The error indicates that it's failing to access a resource with that API group from a cluster API server endpoint. |
Has anyone made any progress on this? It's the last issue I'm running into with my new little AWS based k3s cluster. My issue is exactly as described. I've tried various forms of: I opened up all my SG for All Access across 10.x.x.x.
Checked my DNS settings. Tried raising resource limits for metric-server. Still the same issue. :( Edit: I have re-rolled my cluster and updated to using I was previously using the setup script from here (with the k3s version updated!): https://github.com/sgdan/k3s-test/blob/master/templates/server.j2#L20 I think that removing |
I've disabled the My configuration:
First of all, after an accurate reading of kubernetes documentation I've ended adding
I've disabled the provided So, starting from the official deployment, I've added the followings
And finally, I've added |
Cool, thanks for sharing a walk around. Going to give it a try this weekend. |
@ViBiOh awesome, your solution works, I can finally get some pods and nodes output :)) thanks!! ping on the issue - question remains why it does not work out of the box when installing k3s latest version ? any ideas ? I think this deserve a further investigation.. |
Alas I still can't get it to work (I've just been trying some more). The only thing I haven't yet done from @ViBiOh's instructions is swapped out the default k3s deployment of metrics-server for the official one. Can anyone explain why this makes a difference? I agree with @edenreich - this should work out of the box in k3s. From a lot of reading around and trying things, I can't see that it ever would do - the k3s processes are making the request from the node's network (192.168.x.x in my case) but can't access the cluster's network (10.42.x.x or 10.43.x.x). Has ANYONE actually had success with the default k3s configuration? |
This comment has been minimized.
This comment has been minimized.
I am getting this issue as well with k3s on Hetzner Cloud. Metric Server works great on node1 but timeouts constantly on the other two nodes. The workaround of @ViBiOh works thought, so there is something weird with the initial setup. |
Just wanted to add that I managed to fix this finally. It was a host network issue, where the floating IP that was set for some reason conflicted with the host IP of the node. Using Ubuntu 20.04 and Netplan I had to set the host IP BEFORE the floating IP to not cause some kind of internal routing issue within Kubernetes/k3s. Never managed to figure it out because, but this simple solution fixed all my problems. |
Please avoid using |
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions. |
Thank you, finally got it fixed after a couple of days of headache. |
Hardware
Raspberry Pi 4 8GB RAM (Buster Lite OS)
Version: v1.18.4+k3s1
K3S arguments
Server: --docker --no-deploy=traefik
Agent: --docker
Describe the bug
Fresh installation of k3s, and run
kubectl top nodes
getting error from API 503 service not available.To Reproduce
Install k3s using k3s-ansible with the specified version
Expected behavior
I'd expected to see metrics of nodes
Actual behavior
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
Additional context / logs
Systemd
Server:
The text was updated successfully, but these errors were encountered: