Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed

[wallacepf]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.6
on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v4.24.0
• provider registry.terraform.io/hashicorp/tfe v0.35.0
• provider registry.terraform.io/hashicorp/vault v3.8.0

Affected Resource(s)

• aws_XXXXX

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

data "vault_aws_access_credentials" "ec2-creds" {
  backend = "aws"
  role    = "deploy_ec2"
}

data "vault_aws_access_credentials" "s3-creds" {
  backend = "aws"
  role    = "deploy_s3"
}

data "tfe_outputs" "vault" {
  organization = "my-demo-account"
  workspace    = "hcp-vault"
}

provider "aws" {
  alias  = "ec2"
  region = "us-east-1"

    access_key = data.vault_aws_access_credentials.ec2-creds.access_key
    secret_key = data.vault_aws_access_credentials.ec2-creds.secret_key
}

provider "aws" {
  alias  = "s3"
  region = "us-east-1"

    access_key = data.vault_aws_access_credentials.s3-creds.access_key
    secret_key = data.vault_aws_access_credentials.s3-creds.secret_key
}

provider "vault" {
  address   = data.tfe_outputs.vault.values.vault_public_addr
  namespace = data.tfe_outputs.vault.values.vault_ns

  token = var.vault_token
}

data "aws_availability_zones" "available" {
  filter {
    name   = "zone-type"
    values = ["availability-zone"]
  }
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  providers = {
    aws = aws.ec2
  }

  name = "${var.name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = data.aws_availability_zones.available.names
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = false
  enable_vpn_gateway = false

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

module "ec2_instance" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "~> 3.0"
  providers = {
    aws = aws.ec2
  }
  name = "${var.name}-instance"

  ami                    = "ami-ebd02392"
  instance_type          = "t2.micro"
  key_name               = var.key_name
  monitoring             = true
  vpc_security_group_ids = [module.vpc.default_vpc_default_security_group_id]
  subnet_id              = module.vpc.public_subnets[0]

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

module "s3_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"
  providers = {
    aws = aws.s3
  }

  bucket = "${var.name}-bucket"
  acl    = "private"

  versioning = {
    enabled = true
  }

}

Debug Output

Error: Invalid provider configuration
│ 
│ Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and
│ configure the provider's required arguments as described in the provider documentation.
│ 
╵
╷
│ Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on <empty> line 0:
│   (source code not available)

Expected Behavior

Obtain the credentials via Vault and orchestrate my resources

Steps to Reproduce

1. terraform apply

Important Factoids

I already tried to use another provider version like v3.x but the problem remains. Actually when I try to downgrade, I'm getting a different error:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.74.0"
    }
  }
}

1. terraform init -upgrade

│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given
│ constraints >= 3.63.0, >= 3.72.0, 3.74.0, >= 4.5.0

References

#23209

• #0000

[rrijkse]

Can you try adding provider = aws.ec2 to the availability_zone data source? I am pretty sure the errors you are getting is because you don't have a default provider configured at all. The other option is to remove the alias from one of the providers to act as the default AWS provider.

data "aws_availability_zones" "available" {
  filter {
    name   = "zone-type"
    values = ["availability-zone"]
  }
}

[gdavison]

HI @wallacepf, thanks for submitting this.

@rrijkse is correct about the first error message: Both instances of the aws provider have aliases, but the aws_availability_zones doesn't specify which provider to use, so it tries to use the default aws provider, which isn't declared.

The second error is because the values in data.vault_aws_access_credentials.* are not available yet. Provider configuration occurs before any data sources or resources are called. According to https://developer.hashicorp.com/terraform/language/providers/configuration,

You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

The downgrade error that you're seeing is because one of your modules requires at least v4.5.0.

I'm going to close this issue, since this is expected behaviour. If you run into other problems after updating your configuration, please open a new issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.