Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Issue with IAM role #205

Closed
goelvivek opened this Issue · 15 comments

5 participants

@goelvivek

Hi,
I am using EC2 IAM role. The call to dynamoDB are working fine but once request failed with error

SigningError: Missing credentials in config

It is not happening every time but once it had happened.
Any possible bug with refreshing token ?

The line number from where that error was emitted:

[line] => 201
[pos] => 34
[file] => http.js
[shortStack] => Array
(
[0] => at Socket._onTimeout (net.js:327:8)
)

)

[message] => SigningError: Missing credentials in config
)
@lsegal
Owner

Seems like the SDK timed out when attempting to get credentials from the metadata service, but the stack trace is a little unclear, since it's not a complete trace. We recently changed the way we retry failures from the metadata service in e2a7331 and available in v1.15.0/v2.0.0-rc3 that should continue to retry if this timeout occurs. That said, without more information it's difficult to know if this is a bug in the SDK.

Can you provide a reproduction case or more of the stack trace?

@lsegal
Owner

Closing this since it's unclear what the issue is. Please re-open or create a new issue if you can provide reproduction steps and we will be glad to look into it!

@lsegal lsegal closed this
@goelvivek

Hi, I have updated to the version 1.5.
Still I am facing same issue randomly.

Here is details stack trace.

\n'    at Request.<anonymous> (node_modules/aws-sdk/lib/request.js:262:18)',
      \n'    at Request.callListeners (node_modules/aws-sdk/lib/sequential_executor.js:132:20)',
      \n'    at Request.emit (node_modules/aws-sdk/lib/sequential_executor.js:100:10)',
      \n'    at Request.emitEvent (node_modules/aws-sdk/lib/request.js:536:10)',
      \n'    at Request.failRequest (node_modules/aws-sdk/lib/request.js:497:10)',
      \n'    at Request.<anonymous> (node_modules/aws-sdk/lib/request.js:273:14)',
      \n'---------------------------------------------',
      \n'    at [object Object].handleRequest (node_modules/aws-sdk/lib/http/node.js:59:12)',
      \n'    at MetadataService.request (node_modules/aws-sdk/lib/metadata_service.js:74:10)',
      \n'    at MetadataService.loadCredentials (node_modules/aws-sdk/lib/metadata_service.js:93:10)',
      \n'    at EC2MetadataCredentials.refresh (node_modules/aws-sdk/lib/credentials/ec2_metadata_credentials.js:52:26)',
      \n'    at EC2MetadataCredentials.get (node_modules/aws-sdk/lib/credentials.js:136:12)',
      \n'    at resolveNext (node_modules/aws-sdk/lib/credentials/credential_provider_chain.js:112:15)',
      \n'    at node_modules/aws-sdk/lib/credentials/credential_provider_chain.js:113:11',
      \n'    at node_modules/aws-sdk/lib/credentials.js:138:23'

Error message
Error: Missing credentials in config

@lsegal
Owner

The current version of the SDK is v1.17.3. Can you confirm that you are using this version? Printing AWS.VERSION should tell you this.

@goelvivek

No. I am v1.15.0. As you suggested that IAM patch was included in that version. Will update to new version if it is having any IAM related fix.

@lsegal
Owner

We've not made any changes to the instance metadata service support since 1.15.0, I was just confirming that you were on a later version, since your post said 1.5, not 1.15.0.

I think I might know what the issue is. I will be able to investigate this tomorrow.

@lsegal lsegal reopened this
@lsegal
Owner

Solving this is going to be a little more complicated, but to summarize the issue:

  • The SDK attempts to validate existence of credentials before sending a request, which means it tries to get creds from your metadata service and it fails (temporally).
  • Since the credentials turn up empty, validation fails.
  • The SDK currently does not retry a request if validation or building of a request fails, so this does not trigger any of the retry logic.

The fix to this would be to have all steps retry, but currently retry logic assumes the request is already built, an assumption that is not true if validation triggered a retry. I will be looking into solving this, but in the mean time, you can wrap these calls in your own checks to kick off the request again if it fails:

function makeRequest(req, callback) {
  return req.on('error', function (err) {
    if (err.name === 'CredentialsError') makeRequest(req, callback);
  }).send(callback);
}

makeRequest(s3.listBuckets(), function (err, data) {
  console.log("Done");
});
@goelvivek

@lsegal OK thanks.

@mlogan

@lsegal I just started seeing issues with this error last week. We are still on aws-sdk@1.8.1 (which was the bug fix release you made last time I had a problem with machine role credentials).

I don't have a lot of information yet, but here's what I do have:

  • Happens with dynamodb requests, only after our server has been up for a long time. (Days, not hours.)
  • Once it starts happening, it doesn't stop happening.
  • tcpdump reveals that the broken process is not contacting either the instance metadata service, or dynamodb. That is to say, the error appears to be happening purely within the client. (I verified that a different process that successfully gets data from ddb can be observed via tcpdump to first go to the instance metadata service, and then to 204.246.162.237.https, a dynamodb endpoint, so I'm fairly confident I'm not just missing the requests among all the other noise in the pcap file.)

Sorry, I haven't been able to get any output via AWS.config.logger yet.

I will upgrade to a more recent release, and keep you advised if this error occurs again.

@jfirebaugh

I am also seeing this error during dynamodb requests on a fairly heavily loaded c3.8xlarge. It manifests with the following stack trace:

TimeoutError: Missing credentials in config
    at ClientRequest.<anonymous> (/usr/local/src/app/node_modules/aws-sdk/lib/http/node.js:51:34)
    at ClientRequest.g (events.js:180:16)
    at ClientRequest.EventEmitter.emit (events.js:92:17)
    at Socket.emitTimeout (http.js:1797:10)
    at Socket.g (events.js:180:16)
    at Socket.EventEmitter.emit (events.js:92:17)
    at Socket._onTimeout (net.js:326:8)
    at Timer.unrefTimeout [as ontimeout] (timers.js:412:13)

I haven't been able to check, as @mlogan suggests, that once a given process starts exhibiting the error it will never recover. I do see some processes succeeding (and can manually curl the metatdata service fine) after other processes have failed.

I'm using aws-sdk version 2.0.0-rc9.

@mlogan

Update from me: I haven't seen this bug since my previous post on this issue. We are currently on 1.18.0. Seems that the upgrade fixed it for us.

@lsegal
Owner

I am going to close this since there hasn't been much action in a while, and it seems as though the upgrade has fixed the issue for some users. If you are still having problems, feel free to re-open this or create a new issue.

@lsegal lsegal closed this
@mlogan

I haven't had any issues since my last update, so yes, closing this is fine.

@willwhite

I believe this is still an issue. Let me know if I should open a fresh ticket.

I made an isolated test case for this: https://gist.github.com/willwhite/6afe2d333a002c33413f. The trick to replicating appears to be in adjusting http.globalAgent.maxSockets to a value around 100 (the default is 5 but that's too low for high concurrency applications).

It seems like we need a locking cache around MetadataService.loadCredentials(). To avoid bombarding the metadata service with requests the moment credentials expire.

Happy to work on a PR if there is agreement.

@willwhite

Fresh issue at #445.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.