High latency inside ecs docker with latest versions(From V2.575.0) of aws sdk. It is working till V2.574.0

[manojchowdary27]

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
A clear and concise description of what the bug is.

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?
No Running on ecs docker
Details of the browser/Node.js version
Paste output of npx envinfo --browsers or node -v

SDK version number
Example: v2.466.0

• For browsers, the SDK version number is in the script tag

  src=".../aws-sdk-2.466.0.min.js"

• For Node.js, get SDK version by
  • running command npm list aws-sdk from your root directory
  • printing the output of AWS.VERSION in your code where AWS = require("aws-sdk");
    aws-sdk@2.595.0

To Reproduce (observed behavior)
Steps to reproduce the behavior (please share code or minimal repo)
Install the V2.575.0 or Greater.

1. Inside ec2 query request is taking ~60 ms
2. Inside docker query request is taking ~4500ms
   Expected behavior
   In version 2.24.0 It is working fine in both ec2 and ecs container. It is giving results in ~60ms
   But in the latest versions(From From V2.575.0), It is taking ~4500ms inside the docker.
   Screenshots
   If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[ajredniwja]

Hey @manojchowdary27, thank-you for reaching out to us with your issue can you please share your use case?

[manojchowdary27]

Hey @ajredniwja ,

I'm using dynamo db as a web service in one of my node js application with this aws-sdk package.
Recently I have upgraded to the latest version of aws-sdk.
I have observed the following thing after upgrading (Any version > 2.575).

1. Request dynamo db table from the ec2 directly -> Getting results in 40ms - 60ms
2. Request dynamo db table from ecs docker (which is inside the same ec2 instance) -> Getting results in ~4000ms
   Where as in the versions before 2.574, It is giving results in ~40ms in both ec2 and ecs.

Is there anything to do with ecs or do we need to pass extra config while initiating the sdk?

[ajredniwja]

Hey @manojchowdary27 can you please share minimal code to reproduce or repo.

I wanted to go through the code since there can be different reasons why you are facing latency issues.

[chartrand22]

We experience the same issue with SDK v2.575.0 and up. These bug fixes introduced this issue :

bugfix: IMDS: Signed IMDS workflow
bugfix: IMDS: Type definitions update to signed IMDS workflow

Basically, this was done to tighten up security. (see IMDSv2) The updated flow requires requires a token to be obtained prior to being able to call any metadata endpoints.

From v2.575.0, the SDK is configured to default to the IMDSv2 workflow and, by default, will try three times (with a timeout of one second between attempts) to obtain the required token. If all three attempts fail, the SDK will then fall back to the IDMSv1 workflow, which in your case is successful. Once the credentials are obtained, your code is able to proceed and assume the requested role.

The workaround and/or solutions are :

• use ECS task roles (probably the best solution)
• look into disableFetchToken of the MetadataService (but requires handling refreshing of credentials)
• allow extra hops on the EC2 instance metadata options : http-put-response-hop-limit

[ajredniwja]

@chartrand22 thank-you for explaining that in your comment, I just realized about the same as well, @manojchowdary27 I hope that answers your question, please reach out and re-open if you have any additional questions.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.