-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unable to fetch metrics from node c2: request failed - "403 Forbidden" #990
Comments
add command doesn't work either
|
Sorry, can you print out more logs of metrics-server? I want to know where is the fault |
Thank for your reply. Actually, the logs are the same.
|
And I change service to NodePort. So I clould access by |
From the log, there is no permission to access metrics(Of course, you can't access it directly below.) |
I have been verified the requirements
Nodes must have Webhook authentication and authorization enabled.
Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing --kubelet-insecure-tls to Metrics Server) |
ok, thanks for the verification. What is your environment? EKS?
|
|
Thank you very much. I run it and get returns like this, it seems metrics ok
|
My environment:computer room self-build, which based cloud platform openstack, centos 7。 |
I have the same issue. The only thing that changed was that I upgraded from I am passing |
Ran into the 403 forbidden issue in our eks cluster. Upgrading from 0.5.1 to 0.6.1 and noticed the breaking change in the release notes for 0.6.0 that changes a resource in the cluster role. I added We have multiple clusters, most running 1.19 and one running 1.21 and this made it work in both versions.
|
Thanks @rmendal, I had the same issue. |
Why metrics-server v0.6.x still needs |
I don't understand why? |
@rmendal how do you provision you EKS cluster? If you're using the EKS community module v18 have you made sure you've added the correct security groups or if using the Helm chart set |
@stevehipwell - All of our clusters are provisioned using Terragrunt and the cloud posse eks module. We're using v0.42.1 release. I can confirm it creates the default security group as defined here and we've not altered that at this time. As for the metrics server install I'm using the default v0.6.1 manifest as linked on the release page in this repo, link. The only change I've made is adding the aforementioned |
@rmendal this might be un-related but for Metrics Server to function correctly the control plane needs to be able to reach the node on port 4443 and the nodes need to be able to communicate with each other on port 10250. You can reduce this to just port 10250 by changing the container port and As an aside I'm not sure if the AWS document you linked is correct as I think the outbound 443 was for legacy Metrics Server. |
@stevehipwell so our sec group for the cluster still has an inbound all/all rule but I decided to add an explicit inbound rule for 10250 and then removed However, I then removed the In conclusion, I have no idea what happened to make this work suddenly without the sec group rule and the added line to the cr but it does. If I encounter any other issues in the near term I'll reply here but things suddenly seem to be working as intended. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Adding the line However, it had worked during months, before I encountered the metrics-server failure. I don't understand why this line suddenly disappeared from the ClusterRole/system:metrics-server |
Because this below: Breaking changes |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen |
@ledroide: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I still have this issue with v0.6.4 at my AWS EKS Cluster (Fargate). |
@zentavr for Fargate you need to be using a custom port and you'll need to make sure that you've got that port open in the security group(s). |
I have 2 SGs, one of them is for Control Plane with no rules inside, other one is for the Fargates nodes probably. The service for the metric server listens at 443, while the deployment exposes tcp/10250. which group, which port and which source address should I use for that rule? |
Have this right now:
|
Resolved at my EKS/Fargate cluster:
|
What happened?
I execute this:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
and args:
log shows:
`unable to fetch metrics from node c2: request failed - "403 Forbidden"
`What did you expect to happen?
How to resolve it?
Anything else we need to know?
kubectl describe apiservice v1beta1.metrics.k8s.io
I've been stuck with this problem for a day..
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4", GitTreeState:"clean", BuildDate:"2021", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4", GitTreeState:"clean", BuildDate:"2021", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
/kind support
The text was updated successfully, but these errors were encountered: