AWS_PROFILE not respected for S3 backend when running terraform init/terraform workspace

[Stretch96]

Terraform Version

Terraform v0.11.11

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket  = "example-bucket"
    key     = "example/terraform.tfstate"
    region  = "eu-west-2"
    encrypt = "true"
  }
}

Debug Output

I have created a user, which has no permissions, except the permission to assume the develop role, which has full permissions

Example 1

Running terraform init.
This output is expected, as the user does not have permissions to allow access to the S3 bucket:

Error loading state: AccessDenied: Access Denied
	status code: 403, ...

Example 2

Running AWS_PROFILE=develop terraform init

Error configuring the backend "s3": No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init".

Example 3

Running AWS_SDK_LOAD_CONFIG=1 AWS_PROFILE=develop terraform init

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

These examples are also true with terraform workspace commands

Additional Context

Unfortunately, terraform apply/terraform plan can't be ran with AWS_SDK_LOAD_CONFIG:

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

This makes me think there is a difference in the way that credentials are loaded when using init vs plan/apply

If this can't be reproduced by others, I can provide TRACE logs ... There's just too many redactions to go through, if this can be reproduced elsewhere ...

[abiydv]

Hi @Stretch96,

From the outputs you have posted, seems your aws cli profile is not setup to use your new user. Can you try to setup the profile using aws configure? Once this is ready, you can update your provider block to add the role ARN this user can switch to. The example is mentioned in docs here

[Stretch96]

Hi @abiydv, the aws cli profile is set up correctly, eg:

~/.aws/credentials

[default]
aws_access_key_id=blah
aws_secret_access_key=blah

~/.aws/config:

[profile develop]
region=eu-west-2
cli_follow_urlparam=false
role_arn = arn:aws:iam::123456789012:role/develop
source_profile = default

AWS_PROFILE is respected by provider blocks, however it is not respected by terraform blocks (for the S3 backend) - Without the need to hardcode a role to assume

As shown in the examples, it does work, but AWS_SDK_LOAD_CONFIG needs to be set

[Ashex]

@Stretch96 I've encountered this issue and the root of it is that terraform doesn't seem to support role assumption within credential profiles.

The only workaround I've found is the one mentioned.

[Stretch96]

I've found a way to make terraform crash with AWS_SDK_LOAD_CONFIG set

If you attempt to set the AWS_PROFILE that uses a source_profile that doesn't exist, eg:

# ~/.aws/credentials
[default]
aws_access_key_id=blah
aws_secret_access_key=blah

# ~/.aws/config
[profile test_profile]
region=eu-west-2
role_arn = arn:aws:iam::123456789012:role/test-role
source_profile = oops_typo

Then run:

AWS_SDK_LOAD_CONFIG=1 AWS_PROFILE=test_profile terraform workspace list

Produces the crash output:

2019/05/14 22:25:42 [INFO] Terraform version: 0.11.13
2019/05/14 22:25:42 [INFO] Go runtime version: go1.12
2019/05/14 22:25:42 [INFO] CLI args: []string{"/usr/local/Cellar/terraform/0.11.13/bin/terraform", "workspace", "list"}
2019/05/14 22:25:42 [DEBUG] Attempting to open CLI config file: [redacted]
2019/05/14 22:25:42 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/05/14 22:25:42 [INFO] CLI command args: []string{"workspace", "list"}
2019/05/14 22:25:42 [DEBUG] command: loading backend config file: [redacted]
2019/05/14 22:25:42 [TRACE] Preserving existing state lineage "[redacted]"
2019/05/14 22:25:42 [TRACE] Preserving existing state lineage "[redacted]"
2019/05/14 22:25:42 [INFO] Building AWS region structure
2019/05/14 22:25:42 [INFO] Building AWS auth structure
2019/05/14 22:25:42 [INFO] Setting AWS metadata API timeout to 100ms
2019/05/14 22:25:42 [DEBUG] plugin: waiting for all plugin processes to complete...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1]
goroutine 1 [running]:
github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.unmarshalHandler
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go:119 +0x3e
github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:213 +0x98
github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).Send
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:525 +0x49c
github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.(*EC2Metadata).GetMetadata
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go:28 +0x29a
github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata.(*EC2Metadata).Available(...)
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go:129
github.com/hashicorp/terraform/vendor/github.com/terraform-providers/terraform-provider-aws/aws.GetCredentials
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/terraform-providers/terraform-provider-aws/aws/auth_helpers.go:164 +0x13ea
github.com/hashicorp/terraform/vendor/github.com/terraform-providers/terraform-provider-aws/aws.(*Config).Client
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/terraform-providers/terraform-provider-aws/aws/config.go:278 +0x138
github.com/hashicorp/terraform/backend/remote-state/s3.(*Backend).configure
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/backend/remote-state/s3/backend.go:266 +0xaa6
github.com/hashicorp/terraform/helper/schema.(*Backend).Configure
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/helper/schema/backend.go:80 +0x177
github.com/hashicorp/terraform/command.(*Meta).backend_C_r_S_unchanged
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/command/meta_backend.go:1113 +0x22f
github.com/hashicorp/terraform/command.(*Meta).backendFromConfig
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/command/meta_backend.go:401 +0xe1a
github.com/hashicorp/terraform/command.(*Meta).Backend
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/command/meta_backend.go:88 +0x710
github.com/hashicorp/terraform/command.(*WorkspaceListCommand).Run
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/command/workspace_list.go:44 +0x2ef
github.com/hashicorp/terraform/vendor/github.com/mitchellh/cli.(*CLI).Run
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/vendor/github.com/mitchellh/cli/cli.go:255 +0x1dd
main.wrappedMain
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/main.go:223 +0xb04
main.realMain
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/main.go:100 +0xb4
main.main()
        /private/tmp/terraform-20190311-48945-1cw2gue/terraform-0.11.13/src/github.com/hashicorp/terraform/main.go:36 +0x3b

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

[scho]

I ran into the same issue. Simply specifying the profile in terraform => backend did the trick for me. After that, terraform init|plan|apply works without any permission issues:

terraform {
  backend "s3" {
    # ...
    profile        = "develop"
  }
}

[scalp42]

see #22377 as well

[Puvipavan]

I had the same issue. This is how I fixed it!

According to the documentation, "If you're running Terraform from an EC2 instance with IAM Instance Profile using IAM Role, Terraform will just ask the metadata API endpoint for credentials." Therefore it always taking Instance Role instead of specified profile. So I set environment variable to overwrite aws metadata url.

AWS_METADATA_URL=http://InvalidHost/

So totally you should insert 3 environment variables:

        AWS_METADATA_URL=http://InvalidHost/
        AWS_SDK_LOAD_CONFIG=1
        AWS_PROFILE=develop

If you need to know what happens, then try adding another environment variable for debugging:
TF_LOG=DEBUG

My terraform version is 0.12.21

[bflad]

Multiple fixes for credential ordering, automatically using the AWS shared configuration file if present, and profile configuration handling of the S3 Backend have been merged and will release with version 0.13.0-beta2 of Terraform.

[byron70]

Sorry about the bad post fix etiquette, but thought many may not be able to wait til 0.13.

A work around that worked for me with 0.12.26 was removing all AWS_ environment variables, having all my AWS profile configs in ~/.aws/credentials and explicitly setting the shared_credentials_file pathway along with the profile attribute. I did this in both Windows (c:/users/me/.aws/credentials) and Linux container with success. .

Also, this worked with the S3 backend in one AWS account and provider resources in another (so 2 different aws creds profiles).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.