Error: no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded on v4.0.0

[speller]

On AWS provider version 4.0.0 I'm getting the following error:

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 1, in provider "aws":
│    1: provider "aws" {

Version 3.72.0 works perfectly. I'm using the following provider configuration:

provider "aws" {
  region = var.aws_region
  assume_role {
    duration_seconds = 3600
    session_name = "session-name"
    role_arn = var.aws_deployment_role
  }
}

[gdavison]

Hi @speller, this has been fixed in v4.1.0, which will be released later today.

[thebergamo]

Even using this new version, I still have this error when trying to use the AWS provider.

provider "aws" {
  profile = "default"
  region  = var.aws_region
}

And I can also see that I'm using the latest version of it:

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.1.0...
- Installed hashicorp/aws v4.1.0 (signed by HashiCorp)

Edit:
I see now that it looks like my environment variables are not being picked up by terraform in from version 4.x.x onwards (last working version for me was 3.74.0)

[ashmantak]

Yes i am also facing the same error as above. Is there a workaround

[thebergamo]

Only workaround that I found was to lock version to v3.74.0 you can do it simply by:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.74.0"
    }
  }
}

[ashmantak]

thank you @thebergamo . It worked

[rednevals]

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade

[antonioned]

I see this is closed but I am not sure if it makes sense. For example, I am running some terraform jobs in my CircleCI pipeline and I so far had no issues whatsoever with this, now it seems I need to choose what to use.
Normally, I use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in my CD and profile for local work.

I am thinking of a few things:

1. I can create the ~/.aws/credentials file within the pipeline, with the profile set in it so that it does not fail,
2. I need to break my local env in order to remove profile from the provider and then use env vars. Then again this does not make sense for most of us I presume, since my local ~/.aws/credentials file contains around 15 different profiles.

Maybe I am missing something so if someone has any suggestion, please let me know.

[hectoralicea]

Only workaround that I found was to lock version to v3.74.0 you can do it simply by:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.74.0"
    }
  }
}

where is this file normally located to lock down the version to 3.74? Or what's the file normally called

[thebergamo]

@hectoralicea it would depend mostly on how you organize your code.

Take a look into this docs

[hectoralicea]

@hectoralicea it would depend mostly on how you organize your code.

Take a look into this docs

I think I was able to get past this by adding the text snippet below to the top of the file:
./terraform/modules/base/main.tf
There was no prior terraform {} clause in any of the terraform code I inherited, so I was just able to add it to the top of the file and it worked. (i think. It worked but not sure if this is what did it)

terraform {
  # Can be removed when bug is resolved: https://github.com/hashicorp/terraform-provider-aws/issues/23110
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

[szaffarano]

@hectoralicea
If you want to use 4.x version instead of 3.0 you have to be sure you configure aws either using environment variables or in the .tf file, if you mix both approaches, this version will take just one. More info here

In previous versions of the provider, you could explicitly set profile in the provider, and if the profile did not correspond to valid credentials, the provider would use credentials from environment variables. Starting in v4.0, the Terraform AWS provider enforces the precedence shown above, similarly to how the AWS SDK and AWS CLI behave.

In other words, when you explicitly set profile in provider, the AWS provider will not use environment variables per the precedence shown above. Before v4.0, if profile was configured in the provider configuration but did not correspond to an AWS profile or valid credentials, the provider would attempt to use environment variables. This is no longer the case. An explicitly set profile that does not have valid credentials will cause an authentication error.

[Vacant0mens]

I'm seeing this error message, even with the last 4.3.0 release (similar to this issue).

I set up my local environment from scratch, and made sure my provider config was empty, and configured my aws connection with the aws cli tool (which created ~/.aws/config and ~/.aws/credentials), but I can't get past this error. It looks like a slightly different error than the initial post, but I'm not convinced it's all that different.

with v4.3.0 I get this:

Error: no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed, 
Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout

But with v3.74.3 (with only region in the provider config) I get:

Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers)

I tried adding all sorts of permissions to my aws user that I'm using the token and id for, but plan keeps ending up with the same error no matter what version I'm using.

Is this still an active bug? or did I miss some kind of permissions somewhere?

UPDATE - I gave the user admin access, and still got the same error, so it doesn't seem like a permissions thing.

[Vacant0mens]

I've found that it works if I add the key and id to the provider config, but doesn't work if I use environment variables, or the credentials file, which are supposed to be the next two things in line to check.

[chubin]

I have the same problem, not with terraform, but with my Go program, that uses aws-sdk-v2. Environment variables seems to be not handled properly.

[peterkools]

See the documentation below. There is a good write up about what needs to be changed to resolve this issue.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#changes-to-authentication

[Vacant0mens]

@peterkools I don't see any mention of this error anywhere in that document. And as I mentioned, it only works if I use the provider config, where if there's no provider config, it's supposed to get the credentials from environment variables (as per the precedence order), but it doesn't. I know because I set the environment variables and copied the names straight from the documentation, and also ran the aws cli command to log in and it created the ~/.aws/config like it should've. And neither of those worked.

As @chubin said, It seems like it's only looking in the provider config, and then skipping the environment variables, and going ahead with the connection attempt even if it has nothing.

[Speculor]

I am hitting this too with the latest version of the AWS provider. I have in my versions.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.2.0"
    }
 }

And then in my main.tf:

provider "aws" {
 region = "us-east-1"
}

I have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY defined as variables in Terraform Cloud, and I know they work as they have full admin rights and are used in other Terraform Cloud projects. Strangely, this exact same configuration works on a previous TF Cloud workspace. I get the error:

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found. Please see https://registry.terraform.io/providers/hashicorp/aws for more information about providing credentials. Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout
with provider["registry.terraform.io/hashicorp/aws"]
on main.tf line 11, in provider "aws":
provider "aws" {

[Speculor]

fixed, I had "AWS_ACCESS_KEY_ID" spelled with three "S"'s

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.