Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EKS Pod Identity - Cannot get credentials #3062

Closed
leroylim20 opened this issue Dec 1, 2023 · 16 comments
Closed

EKS Pod Identity - Cannot get credentials #3062

leroylim20 opened this issue Dec 1, 2023 · 16 comments
Assignees
Labels
closed-for-staleness documentation This is a problem with documentation.

Comments

@leroylim20
Copy link

Describe the bug

Using Java SDK 1.12.596, the pod with Pod Identity addon is not able to get AWS credentials via SDK. On the same pod, using AWS CLI v1, it can successfully get its credentials.

On AWS CLI v1

root@f6bd48d86-7x4qn:/# aws --version
aws-cli/1.31.5 Python/3.10.12 Linux/5.10.198-187.748.amzn2.x86_64 botocore/1.33.5
root@f6bd48d86-7x4qn:/# aws ec2 describe-instances
{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "<redacted",
                    "InstanceId": "<redacted>",
                    "InstanceType": "c5.large",

.
.
.

Expected Behavior

Successfully runs the AWS command (E.g. EC2 describe-instances) and prints out its output

Current Behavior

kubectl exec pod/f6bd48d86-7x4qn -c test -it -- bash
root@f6bd48d86-7x4qn:/# curl localhost:8080/version
1.12.596
root@test-f6bd48d86-7x4qn:/# curl localhost:8080/AZ
{"timestamp":"2023-12-01T06:44:30.270+00:00","status":500,"error":"Internal Server Error","path":"/AZ"}
root@f6bd48d86-7x4qn:/# exit
exit
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@b35605f: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4bed011b: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]] with root cause

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@b35605f: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4bed011b: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142) ~[aws-java-sdk-core-1.12.596.jar!/:na]

Reproduction Steps

Run a pod with Java AWS SDK that calls AWS API with pod identity configured

    @GetMapping("/AZ")
    String getAZ() {
        String node_name = System.getenv("NODE_NAME");
        final AmazonEC2 ec2 = AmazonEC2ClientBuilder.defaultClient();

        DescribeInstancesRequest request = new DescribeInstancesRequest().withFilters(new Filter("network-interface.addresses.private-dns-name", Collections.singletonList(node_name)));
        DescribeInstancesResult result = ec2.describeInstances(request);

        String az;
        Optional<Instance> instance;

        if (result.getReservations() != null) {
                instance = result.getReservations().stream().flatMap(reservation -> reservation.getInstances().stream()).findFirst();
                if (instance.isPresent()) {
                    az = instance.get().getPlacement().getAvailabilityZone();
                }
                else {
                System.out.printf(
                        "No instances found with %s%n",
                        node_name);
                return "No instance found";
            }
        }
        else {
            return "API Error";
        }

Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      serviceAccountName: test-serviceaccount
      containers:
      - name: test
        image: <redacted>
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Possible Solution

Fix the credential chain for Pod Identity

Additional Information/Context

No response

AWS Java SDK version used

1.12.596

JDK version used

11

Operating System and version

eclipse-temurin:11

@leroylim20 leroylim20 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 1, 2023
@cademarkegard
Copy link

I'm experiencing the same issue using AWS SDK version 1.12.619

@debora-ito
Copy link
Member

@leroylim20 @cademarkegard sorry for the long silence.

Java SDK 1.x does not support EKS Pod Identity.

2.21.30 is the minimum SDK version that supports EKS Pod Identity - see the list of SDKs and their minimum versions in the Amazon EKS User Guide.

@debora-ito debora-ito added closing-soon This issue will close in 2 days unless further comments are made. and removed needs-triage This issue or PR still needs to be triaged. labels Dec 19, 2023
@debora-ito debora-ito self-assigned this Dec 19, 2023
@cademarkegard
Copy link

hi @debora-ito thanks for the response however it is listed in the CHANGELOG that it is supposed to: https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md#amazon-eks-auth .
Screenshot 2023-12-19 at 10 16 02 AM

Can you please update the documentation if AWS does not have plans on supporting this feature to not mislead others in the future?

@github-actions github-actions bot removed the closing-soon This issue will close in 2 days unless further comments are made. label Dec 19, 2023
@debora-ito
Copy link
Member

Can you please update the documentation if AWS does not have plans on supporting this feature to not mislead others in the future?

Yes, I'll reach out to the EKS documentation team and ask to clarify here the Java SDK v1 does not support EKS Pod Identity.

The changelog entry is the service release note that is automatically sent to all the AWS SDKs when a service feature is released that day. Service features usually don't require changes on the SDK side, but in the case of Pod Identities the Container Credential Provider process required some changes, and those changes were implemented in Java SDK v2 only, they won't be in v1.

@debora-ito debora-ito added the documentation This is a problem with documentation. label Dec 20, 2023
@cademarkegard
Copy link

Thanks so much for the explanation!

@debora-ito debora-ito removed the bug This issue is a bug. label Dec 21, 2023
@daniel-dios
Copy link

daniel-dios commented Feb 20, 2024

Hey it looks like they changed documentation yesterday: <lastBuildDate>Mon, 19 Feb 2024 04:47:18 GMT</lastBuildDate>

I see now EKS is supporting v1: https://docs.aws.amazon.com/pdfs/eks/latest/userguide/eks-ug.pdf#pod-id-minimum-sdk and the error keeps happening.

Wondering if the problem got solved in following versions or what!

cc @rijulGogia

@debora-ito
Copy link
Member

Ok, an update:

Java SDK 1.x doesn't support EKS Pod Identity. The documentation change pointed out by @daniel-dios was made by mistake, and I asked the Documentation team to rectify - there's a banner there now saying

An earlier version of the documentation was incorrect. The AWS SDK for Java v1 doesn't support 
EKS Pod Identity.

@cademarkegard after I asked the Doc team to explicitly add the Java v1 to the list of SDKs and highlight that is not supported, they replied saying the list should include only the SDKs that do support, and they'd like to keep it this way. So hopefully the banner will call attention to it, while it's there.

I'll mark this to autoclose soon. Let me know if there's any follow-up questions.

@debora-ito debora-ito added the closing-soon This issue will close in 2 days unless further comments are made. label Mar 13, 2024
@flozano
Copy link

flozano commented Mar 15, 2024

there is no plan to support EKS Pod identity in AWS SDK V1? rewriting entire services is not an option in most cases.

@github-actions github-actions bot removed the closing-soon This issue will close in 2 days unless further comments are made. label Mar 15, 2024
@debora-ito
Copy link
Member

That's correct, there's no plans to support EKS Pod Identity in V1. This aligns with the V1 deprecation path announced here, V1 will enter maintenance mode July 31, 2024.

@debora-ito debora-ito added the closing-soon This issue will close in 2 days unless further comments are made. label Mar 15, 2024
@flozano
Copy link

flozano commented Mar 15, 2024

that's unfortunate, one would expect that maintenance mode entering in July and being in March, this would be available (general availability ends on July 30, and this was notified just 2 months ago).

That way we would have a little more time to "port" components, while infrastructure can move forward with deploying EKS Pod identity.

Is there any technical difficulty to implement it externally in a custom credentials provider ? Just asking because I may be tempted to attempt implementing it - unless there's a blocker or high technical toll that prevents it.

@github-actions github-actions bot removed the closing-soon This issue will close in 2 days unless further comments are made. label Mar 15, 2024
@flozano
Copy link

flozano commented Mar 22, 2024

Any update on this?

@flozano
Copy link

flozano commented Mar 25, 2024

Any technical insight about why this was discarded (technical impossibility, high difficulty?) would be highly appreciated in order to help me deciding if I can implement myself. Does this require any kind of signature that is not supported in AWS SDK v1 or something of the kin?

Edit: Just to confirm, I'm looking at the code and it seems the only change needed would be to also consider AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE aside from AWS_CONTAINER_AUTHORIZATION_TOKEN. Is that correct? Am I missing something?

@flozano
Copy link

flozano commented Mar 28, 2024

Just to confirm: for us it's working well with this sample project https://github.com/flozano/aws-eks-identity-pod-v1

Given the minimal extent of the changes, the fact that AWS SDK v1 maintenance mode was just recently announced (at least publicly - I'm aware it was announced inside AWS much earlier) and that maintenance mode will not start until July (4 months away), I'm not sure how AWS justifies not supporting this feature.

@flozano
Copy link

flozano commented Mar 28, 2024

I'm wondering what's reasonable to expect from SDK v1 updates, given that in:

https://aws.amazon.com/es/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/#:~:text=Between%20July%2031,%202024%20and,or%20changes%20to%20existing%20services

we are still in General Availability period, and

During this phase, the SDK is fully supported. AWS will provide regular SDK releases that include support for new services, API updates for existing services, as well as bug and security fixes.

just saying.

weeniearms added a commit to weeniearms/aws-sdk-java that referenced this issue Apr 8, 2024
Added support for the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment
variable in the FullUriCredentialsEndpointProvider, which is the
required method of obtaining the authorization token when using the EKS
Pod Identity feature. Moreover, the allowed hosts validation has been
extended to include the ECS and EKS host IPs, which should be allowed
for http (not https) traffic.

Fixes aws#3062
weeniearms added a commit to weeniearms/aws-sdk-java that referenced this issue May 13, 2024
Added support for the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment
variable in the FullUriCredentialsEndpointProvider, which is the
required method of obtaining the authorization token when using the EKS
Pod Identity feature. Moreover, the allowed hosts validation has been
extended to include the ECS and EKS host IPs, which should be allowed
for http (not https) traffic.

Fixes aws#3062
@debora-ito
Copy link
Member

Good news, everyone! We've added support for EKS Pod Identity in v1, starting in SDK version 1.12.746.

I'll reach out to the EKS documentation team again, and ask to update the list here to reflect the change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed-for-staleness documentation This is a problem with documentation.
Projects
None yet
Development

No branches or pull requests

5 participants