-
Notifications
You must be signed in to change notification settings - Fork 733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot run metrics helper with IAM roles mapped to k8s service accounts #663
Comments
@miguelaferreira Were you actually able to get the IAM roles via OIDC working? The documentation here (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html) says that the minimum Go SDK version is 1.23.13, but I believe amazon-vpc-cni-k8s is using 1.21.7 (https://github.com/aws/amazon-vpc-cni-k8s/blob/master/go.mod) |
Yes, I got it working. Instead of the manifest on that same documentation (step 3 in For all other Kubernetes versions) I used this other one that pulls in version v1.5.4. I've made a terraform module that is able to upgrade the CNI plugin that is installed by default in EKS, and another one that sets up the IAM side of things. |
@miguelaferreira There is an issue with ip rules going missing in v1.5.4 (#641), please try the v1.5.5 release candidate instead. |
I've tried that version @mogren but I still get the same output. With network policy blocking access to the EC2 metadata endpoint pod complaints it needs that access:
Without the network policy blocking access to the EC2 metadata endpoint pod assumes the role of the worker node and then complaints because it does not have access to cloudwatch:
|
@miguelaferreira Oh, did you add that permission though? It's not available in the managed CNI policy by default. See https://docs.aws.amazon.com/eks/latest/userguide/cni-metrics-helper.html#install-metrics-helper for details |
@mogren I'm not sure what permission you are referring to. But if that's the policy to allow the pod to call Using the role for the SA I have to block the access to the EC2 metadata, otherwise the pod assumes the role of the worker node ( Does that clarify the problem? |
Ah, thanks @miguelaferreira for the explanation. This requires some more work from our side. |
@mogren is there any progress towards supporting running the metrics helper with IAM roles mapped to k8s service accounts? |
@miguelaferreira Sorry, not yet, but thanks for pinging me about it. Similar changes should be done to the ipamd pod ( |
@mogren I was checking back on this issue when I re-read your comment. I'm not sure what needs to change in the ipamd pod but I can confirm it works perfectly with IAM roles mapped to k8s service accounts. I have the metadata endpoint blocked on my cluster and the ipamd pods are using the role I assign to them.
|
@miguelaferreira Have you applied the same above changes (for AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE) to the CNI metrics helper |
@jaypipes I'm not sure I understand what you are asking. But the way I have been doing this is to annotate a service account and then the pod spec gets extended with these extra env vars. I have done this consistently with several deployments in my cluster. |
@miguelaferreira yes, sorry for being unclear. @mogren I believe I have found the source of this problem. Note that the CNI metrics helper instantiates the AWS SDK Session object differently than ipamd. Here is the CNI metrics helper instantiating its Publisher's session: amazon-vpc-cni-k8s/pkg/publisher/publisher.go Lines 92 to 108 in 71538ac
and here is where the Metrics client ends up instantiating its session: amazon-vpc-cni-k8s/pkg/ec2wrapper/ec2wrapper.go Lines 32 to 40 in 71538ac
Note that in the latter case, we call which queries IMDS for the instance-identity/document path. I think something to do with GetInstanceIdentityDocument() and the different between the publisher and metrics client is the source of the issue here. /cc @micahhausler |
As @jaypipes mentioned we need to look at the IPAMD and metrics helper code to understand how the session is setup. That should clarify why the behavior is different. |
Seems related to 1287. Any updates? |
@jayanthvn Please let us know if there are any updates to this issue. EKS cluster deployed with v1.9.0 amazon-k8s-cni and pods blocked from IMDS. cni-metrics-helper deploys, but reports failure for EC2MetadataRequstError. cni-metrics-helper will be useful to monitor the enis and ips associated with EKS deployment. Thank you.
|
@kumarpmd Hi, we have released a private image to test this fix. Could you try with this cni-metrics-helper image tag: v1.10.2-rc1 ? You will also need to specify AWS_CLUSTER_ID as below and use an IRSA so that Region field is auto-injected
Refer this readme from above linked PR for guidance: https://github.com/aws/amazon-vpc-cni-k8s/blob/af780320a81bab5fca5473ce22c61965aa18141a/cmd/cni-metrics-helper/README.md |
Thank you, @cgchinmay. cni-metrics-helper:v1.10.2-rc1 with the AWS_CLUSTER_ID was deployed in a cluster with pods blocked from IMDS, and cni-helper was able to generate metrics.. Thank you again. Should the version of vpc-cni match cni-metrics-helper? Tested cni-metrics-helper:v1.10.2-rc1 with vpc-cni v1.9.0 in eks 1.20, and cni's svc account aws-node. Just deployed role and bindings for aws-node. Let me know if this is of concern. |
Thanks for confirming @kumarpmd , no you dont need to. Changing only cni-metrics-helper should be enough. |
PR 1715 is merged and will be part of 1.10.2 release. EKS documentation will updated after release and GitHub readme is updated. |
|
I've setup my EKS cluster to block pod access to the EC2 metadata endpoint and instead obtain IAM policies via roles mapped to service accounts (via OpenID Connect).
Turns out that the cni metrics helper wants to reach that endpoint. Since I'm blocking pod access to the EC2 metadata with (calico) network policies, I've allowed that one pod (metrics helper) to reach the endpoint. What happens next is that, since the pod can reach the EC2 metadata endpoint, it assumes the worker role instead of the role I created for it.
I'm stuck in between the setup with IAM roles mapped to k8s service accounts and running the metrics helper. Is there a way to have both?
The text was updated successfully, but these errors were encountered: