Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java SDK does not support EKS IAM for service accounts #1470

Closed
starchx opened this issue Oct 13, 2019 · 20 comments
Closed

Java SDK does not support EKS IAM for service accounts #1470

starchx opened this issue Oct 13, 2019 · 20 comments
Labels
bug This issue is a bug.

Comments

@starchx
Copy link

starchx commented Oct 13, 2019

EKS IAM Service Account Role introduces a new environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and based on the documentation on these two pages, the Java SDK should use "AWS_WEB_IDENTITY_TOKEN_FILE" for credentials if exists.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

I have tried the latest Java SDK 2.9.19 and it doesn't seem to work. There is a similar discussion on DotNet SDK at here:

aws/aws-sdk-net#1413

I couldn't find docs saying "AWS_WEB_IDENTITY_TOKEN_FILE" is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.

Expected Behavior

Based on the EKS doc link above, Java SDK should recognize the environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.

Current Behavior

I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run aws s3 ls and it worked, which means the token is correct.

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 467C66AAEAD354A9)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listBuckets(DefaultS3Client.java:2035)
        at com.example.s3.S3BucketOps.main(S3BucketOps.java:63)

Steps to Reproduce (for bugs)

The code I am using to test is from sample S3 code at here:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java

I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn't work.

Your Environment

  • AWS Java SDK version used: 2.9.19
  • JDK version used: 1.8
  • Operating System and version: Linux running from a container (pod on EKS)
@endre-synnes
Copy link

endre-synnes commented Oct 17, 2019

Having the same issue using the AWS Java SDK version: 1.11.653.

  • JDK version used: 11
  • Operating System and version: Linux running from a container (pod on EKS)
  • Kubernetes version 1.14

According to the documentation this version of the SDK should also work fine. My application uses the role of the node instead of the one I have added through a service account.

@starchx
Copy link
Author

starchx commented Oct 18, 2019

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

starchx added a commit to starchx/amazon-eks-user-guide that referenced this issue Oct 18, 2019
Customers got confuse and thought once they upgraded the SDK to the listed version or above, the web identity token would be used for exchanging AWS credentials without any code changes. However, it is not the case due to WebIdentityTokenFileCredentialsProvider is not in the default credential provider chain for certain SDKs, for example:

     Java SDK - aws/aws-sdk-java-v2#1470
     DotNet SDK - aws/aws-sdk-net#1413
@endre-synnes
Copy link

@starchx Thank you! That solved the issue😄 Hope that the WebIdentityTokenFileCredentialsProvider will be added as a default soon!

@dagnir
Copy link
Contributor

dagnir commented Oct 23, 2019

Hi all, this should be part of the default chain. We will prepare a change for this.

@dagnir dagnir added the bug This issue is a bug. label Oct 23, 2019
@thundergolfer
Copy link

@endre-synnes I am also experiencing the same problem despite the documentation saying it should work. We should file an issue at https://github.com/aws/aws-sdk-java/issues

@zoewangg
Copy link
Contributor

zoewangg commented Nov 7, 2019

Fixed via #1501 and released as part of 2.10.11

@zoewangg zoewangg closed this as completed Nov 7, 2019
@thereisnospoon
Copy link

thereisnospoon commented Feb 25, 2020

Hi,
It still doesn't work for me with 2.10.57. The sdk uses node role instead of service account role.

UPDATE
It works. I just had to add

      securityContext:
        fsGroup: 65534

to the pod spec in order to make sure containers have access to the service token file.

@imcheck
Copy link

imcheck commented Mar 19, 2020

Hi, It still doesn't work for me with 2.10.72.
It seems that java sdk doesn't get credentials from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN envs.
(FYI, I already added security context to pods to be able to access the token file)

@thereisnospoon
Copy link

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

@jiang-wei
Copy link

jiang-wei commented Apr 20, 2020

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

It just works :)

@gargkshitiz
Copy link

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

@jiang-wei
Copy link

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

S3Client s3 = S3Client.builder()
                              .region(region)
                              .build();

and include sts with the latest version from maven repo, today it is 2.11.14

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

@gargkshitiz
Copy link

Thank you Jiang. This was conflicting with aws sdk jar I was using. Latest versions of sdk and sts helped me.

@campbel
Copy link

campbel commented Jun 10, 2020

This is an interesting discussion and somewhat related to a use case I am trying to figure out. My service needs to access resources in a different AWS account from EKS, so I want to use profiles and assume role using a config like this:

[profile eks-account-role]
role_arn = <eks-account-role-arn>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile other-account-role]
role_arn = <other-account-role-arn>
source_profile = eks-account-role
role_session_name = x-account-session

Using the following code to access the STS client (for other purposes than assuming roles).

stsClient = StsClient.builder().build();

Then I set the environment variable AWS_PROFILE=other-account-role in the container.

But when I go to run it, the service is still running as the eks-account-role. I'm assuming the default credential provider chain will pick up the AWS profile defined in the config and use that method for authentication, but it doesn't seem to be happening. However, when I try this with the AWS CLI it works as expected (I have properly setup the other role to be assumed by the eks role).

I have found a workaround where if I specify to use profile credentials explicitly it works as expected.

stsClient = StsClient.builder()
                .credentialsProvider(ProfileCredentialsProvider.create())
                .build();

Now this isn't a big deal as it is a small amount of additional code, but I am curious what the SDK is doing under the hood and why it is different from the CLI. A couple things to consider, EKS seems to be injecting two environment variables into the container that could be effecting the behavior AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. However, I am running the CLI from within the container with the same variables present.

Thoughts?

@rod-dvla
Copy link

rod-dvla commented Dec 9, 2020

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

It just works :)

Surprised that this works, but it does. Any idea why the separate declaration of sts is required?

@StealthyDev
Copy link

StealthyDev commented May 28, 2021

I still get an error trying to connect to Secrets Manager from EKS using Service Account role.
I'm using sdk v2.16.52.

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 13e02bf6-082c-47a2-ace4-900b02792b72, Extended Request ID: null), ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.secretsmanager.DefaultSecretsManagerClient.getSecretValue(DefaultSecretsManagerClient.java:1098) at com.hlpp.csw.util.SecretManagerUtil.getSecretFromSM(SecretManagerUtil.java:51) at com.hlpp.csw.config.ShiroCustomEnvironment.setIniOverrides(ShiroCustomEnvironment.java:23) at com.hlpp.csw.config.ShiroCustomEnvironment.init(ShiroCustomEnvironment.java:15) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:45) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:40) at org.apache.shiro.web.env.EnvironmentLoader.createEnvironment(EnvironmentLoader.java:313) at org.apache.shiro.web.env.EnvironmentLoader.initEnvironment(EnvironmentLoader.java:139) at org.apache.shiro.web.env.EnvironmentLoaderListener.contextInitialized(EnvironmentLoaderListener.java:58) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:692) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1023) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:824) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1611) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:829) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:433) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)

@llnformer
Copy link

still happens

SDK version: software.amazon.awssdk:ssm:2.17.19
JDK version used: Java 11
Operating System and version: openjdk:11.0.11-jdk-slim
EKS version: 1.21

only after I added software.amazon.awssdk:sts:2.17.19 fixed

@itowk
Copy link

itowk commented Aug 1, 2022

Hi,
We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:

  • aws-java-sdk-core (v1.12.239)
  • aws-java-sdk-sns (v1.11.125)

@madorb
Copy link

madorb commented Aug 26, 2022

@itowk. i don't know if it'll solve your issue, but definitely don't use misaligned versions of the sdk. there's a bom for a reason.

@maufranchini
Copy link

Hi, We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:

  • aws-java-sdk-core (v1.12.239)
  • aws-java-sdk-sns (v1.11.125)

Adding the latest v1 version (1.11.1034 today's date) for the aws-java-sdk-sts dependency solved my issue (no code needed to be touched using the method DefaultAWSCredentialsProviderChain(). I hope it helps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests