provider-aws: Intermittent Race Condition with Ec2 Metadata service/credentials

[jrnt30]

Issue:

While plan or applying our infrastructure when using the EC2 Metadata Service for TF AWS credentials we are seeing intermittent issues with our larger TF stack where we receive varying numbers of No valid credential sources found for AWS Provider. error messages.

A few things to note:

• Doesn't ever happen on some of our smaller TF stacks
• If we pass the credentials via environment variables explicitly it works every time
• Configuring the credentials file explicitly on the node it will work fine
• If we use the IAM Role/Metadata provided credential provider it works intermittently (maybe 20% of the time)
• If we limit the parallelism with the IAM Role credential provider it works most of the time as well when reduced down to 3. 5 and 7 still fail relatively frequently
• Get the "No valid credential sources found for AWS Provider." error on both Plan and Execute
• Increasing log level to "TRACE" slows things down enough for us to prevent the issue from occurring :)

My guess is that there is an API rate limiting occurring when using the IAM Role provided credentials with the default parallelism, curious if others have run into similar issues.

Terraform Version:

terraform-0.6.14

Affected:

• AWS Provider configuration w/ IAM Roles/Metadata service

Debug Output (Stripped to just relevant counts):

Count for [./credentials_file_2016-04-18-171349-BROKE - Building AWS auth structure]  24
Count for [./credentials_file_2016-04-18-171349-BROKE - EC2 Metadata service found, adding EC2 Role Credential Provider]  24
Count for [./credentials_file_2016-04-18-171349-BROKE - EC2 Metadata service not found, not adding EC2 Role Credential Provider]  0

Panic Output:

No panic output, however do get:

* No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Expected Behavior

Reduce contention on Metadata service calls or handle/retry responses

[jrnt30]

@phinze Per our conversation in chats today. Thanks for your help as well

[jrnt30]

@phinze I was able to confirm that the metadata service does start sending 429s when it's being hit heavily. In my concurrency test it seems to break down after 25 concurrent connections are made.

[Solisol]

Is there a fix for this coming? friendlybump

[dcarley]

We've seen this too. I'm not clear though whether there is currently none or some retry and backoff in place.

GetCredentials() passes an aws.Config which uses client.DefaultRetryer. I can't tell for sure whether it will retry on 429 responses or whether MaxRetries needs to be set for ec2metadata.

[sthulb]

I've experienced this too, but not with the metadata auth, but with a API key.
tf.txt

[sstarcher]

I was wondering why my CI was intermittently failing, but I could not reproduce locally.

[sstarcher]

@ajmath have you experienced this in your setup?

[yogeshnath]

I'm seeing the same issue. If I run my larger stack, it fails with the same error message (No valid credential sources found for AWS Provider).

However, the number of times I see this message varies with every run. I'm suspecting it to be some race condition in getting the iam-role from the metadata....

[seanjoo]

I am having the same issue too. Is this going to be addressed anytime soon?

[jszwedko]

Anecdotally this issue has gotten worse in 0.7.x (we worked around it in 0.6.x by reducing the parallelism). Even when setting -parallelism=1with 0.7.x we are still encountering it.

[hugoduncan]

I'm seeing this behaviour with 0.8.5.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.