Terraform Remote State Fails: AuthorizationHeaderMalformed: The authorization header is malformed

[johneipe]

Hi there,

I'm trying to try the remote state management with s3 and locking using dynamoDB. But it fails with error:
Error inspecting states in the "s3" backend: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

Terraform Version

terraform -version
2018/09/12 18:17:44 [INFO] Terraform version: 0.11.8
2018/09/12 18:17:44 [INFO] Go runtime version: go1.10.3
2018/09/12 18:17:44 [INFO] CLI args: []string{"/usr/local/Cellar/terraform/0.11.8/bin/terraform", "-version"}
2018/09/12 18:17:44 [DEBUG] Attempting to open CLI config file: /Users/johne/.terraformrc
2018/09/12 18:17:44 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/09/12 18:17:44 [INFO] CLI command args: []string{"version", "-version"}
Terraform v0.11.8
2018/09/12 18:17:44 [DEBUG] checking for provider in "."
2018/09/12 18:17:44 [DEBUG] checking for provider in "/usr/local/Cellar/terraform/0.11.8/bin"

2018/09/12 18:17:44 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2018/09/12 18:17:44 [DEBUG] plugin: waiting for all plugin processes to complete...

Terraform Configuration Files

variable "region" {
  default = "us-east-1"
}

# defining a provider: aws
provider "aws" {
  shared_credentials_file = "~/.aws/config"
  profile = "dev"
  region = "${var.region}"
}

# terraform state file setup
# create an S3 bucket to store the state file in
resource "aws_s3_bucket" "terraform-state-storage-s3" {
  bucket = "jeipe-remote"
  versioning {
    enabled = true
  }
  lifecycle {
    prevent_destroy = true
  }
  tags {
    Name = "S3 Remote Terraform State Store"
  }
}
# create a dynamodb table for locking the state file
resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {
  name = "jeipe-state-lock-dynamo"
  hash_key = "LockID"
  read_capacity = 20
  write_capacity = 20

  attribute {
    name = "LockID"
    type = "S"
  }

  tags {
    Name = "DynamoDB Terraform State Lock Table"
  }
}

# defining terraform state to be backend s3
terraform {
  backend "s3" {
  region = "us-east-1"
  encrypt = true
  bucket = "jeipe-remote"
  dynamodb_table = "jeipe-state-lock-dynamo"
  key = "terraform.tfstate"
  }
}

resource "aws_security_group" "instance" {
  name = "terraform-example-instance"
  ingress {
    from_port = "${var.server_port}"
    to_port = "${var.server_port}"
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Debug Output

Last few lines from the logs

2018/09/12 17:16:52 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /?prefix=env%3A%2F HTTP/1.1
Host: terraform-remote-state-storage-s3.s3.amazonaws.com
User-Agent: aws-sdk-go/1.14.31 (go1.10.3; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8
Authorization: AWS4-HMAC-SHA256 Credential=AKIAI2M*****H66NDZA/20180912/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a2c2358d87f2f78fb8130**********354c0fef88301
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996f*********95991b7852b855
X-Amz-Date: 20180912T114652Z
Accept-Encoding: gzip


-----------------------------------------------------
2018/09/12 17:16:54 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 12 Sep 2018 11:46:53 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: GXxSZfF6R+gjZNy*******LSasoYqV7ztiGVnhUIcE=
X-Amz-Request-Id: 039B09C373FAFEAB


-----------------------------------------------------
2018/09/12 17:16:54 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'</Message><Region>us-west-2</Region><RequestId>039B09C373FAFEAB</RequestId><HostId>GXxSZfF6R+gjZN*********V7ztiGVnhUIcE=</HostId></Error>
2018/09/12 17:16:54 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjects failed, not retrying, error AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
	status code: 400, request id: 039B09C373FAFEAB, host id: GXxSZfF6R+gjZNyPO***********eLSasoYqV7ztiGVnhUIcE=
2018/09/12 17:16:54 [DEBUG] plugin: waiting for all plugin processes to complete...
Error inspecting states in the "s3" backend:
    AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
	status code: 400, request id: 039B09C373FAFEAB, host id: GXxSZfF6R+gjZNyP**************SasoYqV7ztiGVnhUIcE=

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

Expected Behavior

Should create security groups and keep the state in s3 and maintain locks using dynamodb

Actual Behavior

Fails to create the backend state

Steps to Reproduce

terraform init -backend-config="access_key=*******" -backend-config="secret_key=************"

References

Similar to #2774 but the bucket name is unique.

[mildwonkey]

The first think I would check is that your s3 bucket and dynamoDB table are in us-east-1.

The error suggests that the s3 bucket you are referencing for terraform remote state is in us-west-2. The terraform block must reference the region of you s3 bucket and dynamoDB table, not necessarily the region that you are building resources in.

So, for example, your configuration might look like this (provided that the s3 bucket and dynamoDB table are in us-west-2):

terraform {
  backend "s3" {
  region = "us-west-2"
  encrypt = true
  bucket = "jeipe-remote"
  dynamodb_table = "jeipe-state-lock-dynamo"
  key = "terraform.tfstate"
  }
}

provider "aws" {
  shared_credentials_file = "~/.aws/config"
  profile = "dev"
  region = "us-east-1"
}

[johneipe]

Oh. I thought the latest release provides a feature to create the bucket and the table based on the configuration given. This has to be created in advance and then referenced?

[mildwonkey]

Correct, terraform does not automatically create the resources required for remote state. Those need to be created before use.

[johneipe]

@mildwonkey
Okay that means the below entries are not needed right?

resource "aws_s3_bucket" "terraform-state-storage-s3" {
  bucket = "jeipe-remote"
  ....
}
# create a dynamodb table for locking the state file
resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {
  name = "jeipe-state-lock-dynamo"
  ...
}

Anyways I created the table and bucket in us-east-1 and then tried the script with and without the above entries. But I'm faced with the same error.

Error inspecting states in the "s3" backend:
AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

Not sure from where the us-west-2 is popping up.

[mildwonkey]

Sounds like something on your system is telling AWS to point to us-east-1 by default.
Did you intend for the backend to use the same credentials as the provider? You'd need to add the profile and shared_credentials_file to the terraform block.

The AWS CLI can be really helpful for troubleshooting - can you run aws s3 ls jeipe-remote? (It looks like your terraform configuration is using default AWS credentials so in theory it'll use the same credentials terraform is using).

You might want to check for any unexpected AWS environment variables (AWS_REGION or AWS_DEFAULT_REGION_

And in your credentials file" Are you AWS credentials defaulting to us-east-1? Something like this?

[default]
region = us-east-1

[dev]
region = us-east-1

[johneipe]

I found the problem; I had different credentials in ~/.aws/credentials and in ~/.aws/config.

Now that I have cleaned it up it is working fine but still while running the init and plan it's asking for region eventhough I have mentioned it in config.

~/.aws/config

[dev]
region = "us-east-1"

~/.aws/credentials

[default]
aws_access_key_id =****
aws_secret_access_key =****
[dev]
aws_access_key_id =****
aws_secret_access_key =****

main.tf

# defining a provider: aws
provider "aws" {
  profile = "dev"
}

# defining terraform state to be backend s3
terraform {
  backend "s3" {
  encrypt = true
  bucket = "jeipe-remote"
  dynamodb_table = "jeipe-state-lock-dynamo"
  key = "terraform.tfstate"
  }
}

During execution,

$ terraform init

Initializing the backend...
region
  The region of the S3 bucket.

  Enter a value:

It shouldn't be asking right?

[mildwonkey]

Progress!

You need to specify a region or profile in your terraform block or (I think) in ~/.aws/credentials. The backend configuration does not use the same value from provider and it appears that the AWS SDK is prioritizing the config in $HOME/.aws/credentials over that in ~/.aws/config.

Hope this helps!

[bflad]

I believe setting the AWS_SDK_LOAD_CONFIG=1 environment variable can help if you're trying to load the region from the configuration file. https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/sessions.html

Admittedly, this is not well documented in the backend/provider documentation and in the future we may consider enabling loading the configuration by default (in a major version upgrade).

[johneipe]

@mildwonkey @bflad
Thanks.
I dropped the whole config thing. I'm using credentials file to manage just the credentials and everything else including region goes into terraform files. That way it's simple now.

[BastianM3]

I ran into this issue when I accidentally set a data block to point at a region that didn't actually contain my state bucket (multi-region configuration).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.