Unable to load credentials from service endpoint when using IAM role #1398

palharsh · 2017-11-23T14:19:10Z

aws-java-sdk-sqs version 1.11.170

Using an aws task with IAM role having the right access to an SQS queue. It worked all fine but without any change in any config from our side we started getting an exception like

com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
    at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:180)
    at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:159)
    at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
    at com.amazonaws.auth.ContainerCredentialsProvider.getCredentials(ContainerCredentialsProvider.java:57)
    at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:51)
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:110)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
    at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716)
    at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380)
    at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356)
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$AsynchronousMessageListener.run(SimpleMessageListenerContainer.java:277)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
    at sun.net.www.http.HttpClient.New(HttpClient.java:308)
    at sun.net.www.http.HttpClient.New(HttpClient.java:326)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
    at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:47)
    at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:106)
    at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121)

We are getting this exception consistently even after relaunching the service.
The same docker image works fine with similar IAM role on a different container instance.

The text was updated successfully, but these errors were encountered:

dagnir · 2017-11-28T19:21:47Z

The error indicates that the SDK could not connect to the default credentials host at 169.254.170.2. Can you try the cURL example from this page from within the failing instance to see if it works outside of the SDK?

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

palharsh · 2017-11-30T15:38:34Z

The error went away the next day, again without us doing anything. So did not get chance to investigate more. But it persisted for more than a day. But i would really like to know what happen ed, so that it does not happen in our production environment (it was in dev)

dagnir · 2017-11-30T17:07:01Z

Good to see you're no longer experiencing the issue. Unfortunately as stated above, the only thing we can glean from the stacktrace is that the client was unable to connect to the standard credentials endpoint; without any more information it's difficult to track down why. If you haven't already, it might be helpful to open a ticket in the support center to see if it may have been a problem with how ECS created the problematic containers.

varunnvs92 · 2017-12-06T05:06:07Z

Feel free to reopen if you have more questions.

kfh · 2017-12-07T07:55:07Z

Please reopen. We are seeing it a lot in our api's now. Using aws-java-sdk version 1.11.235.

For reference:

Unable to load credentials json.stack_trace:"com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:110) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2186) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2162) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executePutItem(AmazonDynamoDBClient.java:1466) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.putItem(AmazonDynamoDBClient.java:1442) at com.amazonaws.services.dynamodbv2.document.internal.PutItemImpl.doPutItem(PutItemImpl.java:85) at com.amazonaws.services.dynamodbv2.document.internal.PutItemImpl.putItem(PutItemImpl.java:63) at com.amazonaws.services.dynamodbv2.document.Table.putItem(Table.java:168) at com.tomra.csw.api.clearing.reporting.repository.ReportingPeriodRepository.saveClearingReportingPeriod(ReportingPeriodRepository.java:122) at com.tomra.csw.api.clearing.reporting.repository.ReportingPeriodRepository.createReportingPeriod(ReportingPeriodRepository.java:77) at com.tomra.csw.api.clearing.reporting.service.ReportingPeriodService.createReportingPeriod(ReportingPeriodService.java:72) at com.tomra.csw.api.clearing.reporting.service.ReportingPeriodService.generateAndUploadR2Report(ReportingPeriodService.java:245) at com.tomra.csw.api.clearing.reporting.endpoint.ExchangeForChangeEndpoint.genenrateR2Rreport(ExchangeForChangeEndpoint.java:37) at spark.RouteImpl$1.handle(RouteImpl.java:61) at spark.http.matching.Routes.execute(Routes.java:61) at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:126) at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) at org.eclipse.jetty.server.Server.handle(Server.java:517) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745)Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:110) at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ... 42 common frames omitted"

varunnvs92 · 2017-12-07T18:47:24Z

I don't think this issue is on SDK end as this code has not changed recently. As dagnir@ pointed out, the issue might be on the service end. You can reach out to the service team through console support.

You can provide wire logs to us. That might help in narrowing down the issue. Make sure there is no sensitive data.

dagnir · 2017-12-21T19:27:43Z

Closing due to no response. If you observe this issue again, can you try the cURL command from this page?

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

If you're unable to fetch credentials using that command then it's some other issue with ECS or the container.

sumithtomy · 2018-08-30T14:02:45Z

You can open a tunnel from your local machine to the amazon instance.
ssh -N ec2-user@<your-ip> -i <your-key-pair>.pem
if required use port forwarding with -L option too.
ssh -N -L <your-port> ec2-user@<your-ip> -i <your-key-pair>.pem

JJonesAtAvaya · 2019-09-17T19:37:35Z

I have this same issue...
I'm trying to use an IAM role from an ECS container to access SQS
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'simpleMessageListenerContainer' defined in class path resource [org/springframework/cloud/aws/messaging/config/annotation/SqsConfiguration.class]: Invocation of init method failed; nested exception is com.amazonaws.AmazonServiceException: Internal Server Error (Service: null; Status Code: 500; Error Code: null; Request ID: null)

I've tried many things, but am using this for connecting.
I have been unable to connect using any sample code I've found so far


  @Bean
  @Primary
  public AmazonSQSAsync amazonSQSAsyncClient() {
    return AmazonSQSAsyncClientBuilder.standard()
        .withCredentials(amazonAWSCredentials())
        .build();
  }

  @Bean
  @Primary
  public AWSCredentialsProvider amazonAWSCredentials() {
    log.info("HELLO Trying AWS authentication via instance profile credentials provider");
    return new InstanceProfileCredentialsProvider(false);
  }

I tried to run the curl command as suggested earlier, but there is no environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
populated in the container

shambhand · 2020-09-29T09:41:42Z

Facing the same issue intermittently with AWS SDK v1.11.788, adoptopenjdk11 on kubernetes cluster trying to connect using Kube2Iam

ffeltrinelli · 2022-04-08T18:38:47Z

Hi! I talked about this issue and described our custom solution in this article.

palharsh closed this as completed Nov 23, 2017

palharsh reopened this Nov 23, 2017

varunnvs92 added the Question label Nov 24, 2017

dagnir added the waiting-reply label Nov 28, 2017

dagnir removed the waiting-reply label Nov 30, 2017

varunnvs92 closed this as completed Dec 6, 2017

varunnvs92 reopened this Dec 7, 2017

dagnir closed this as completed Dec 21, 2017

srchase added guidance Question that needs advice or information. and removed Question labels Jan 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to load credentials from service endpoint when using IAM role #1398

Unable to load credentials from service endpoint when using IAM role #1398

palharsh commented Nov 23, 2017 •

edited

Loading

dagnir commented Nov 28, 2017

palharsh commented Nov 30, 2017

dagnir commented Nov 30, 2017

varunnvs92 commented Dec 6, 2017

kfh commented Dec 7, 2017

varunnvs92 commented Dec 7, 2017

dagnir commented Dec 21, 2017

sumithtomy commented Aug 30, 2018 •

edited

Loading

JJonesAtAvaya commented Sep 17, 2019

shambhand commented Sep 29, 2020

ffeltrinelli commented Apr 8, 2022

Unable to load credentials from service endpoint when using IAM role #1398

Unable to load credentials from service endpoint when using IAM role #1398

Comments

palharsh commented Nov 23, 2017 • edited Loading

dagnir commented Nov 28, 2017

palharsh commented Nov 30, 2017

dagnir commented Nov 30, 2017

varunnvs92 commented Dec 6, 2017

kfh commented Dec 7, 2017

varunnvs92 commented Dec 7, 2017

dagnir commented Dec 21, 2017

sumithtomy commented Aug 30, 2018 • edited Loading

JJonesAtAvaya commented Sep 17, 2019

shambhand commented Sep 29, 2020

ffeltrinelli commented Apr 8, 2022

palharsh commented Nov 23, 2017 •

edited

Loading

sumithtomy commented Aug 30, 2018 •

edited

Loading