Terraform error operation error S3: ListObjectsV2, https response error StatusCode: 403

[simonweil]

Terraform Version

Terraform v1.6.3
on darwin_arm64

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket         = "state-bucket-name"
    key            = "<AWS-ACCOUNT-ID>/<accountname>/<env>/terraform.tfstate"
    dynamodb_table = "terraform-state-lock-table-name"
    region         = "<theregion>"
    encrypt        = true

    assume_role = {
      role_arn     = "<rolename-to-assume>"
      external_id  = "<external_id>"
      session_name = "test"
    }
  }
}

output "test" {
  value = "test"
}

Debug Output

https://gist.github.com/simonweil/e023ac03b6966a05f712c93573e86e5b

Expected Behavior

The terraform init should succeed like for all previous terraform versions.

Actual Behavior

terrafrom init fails accessing the backend with the following error

Initializing the backend...
╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "terraform-state-bucket-name": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: Access Denied

Steps to Reproduce

1. create a backend s3 bucket
2. create an assumable role with the following policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StateBucketObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObjectAcl",
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::terraform-state-bucket-name/<acctid>/*"
        },
        {
            "Sid": "StateBucketList",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::terraform-state-bucket-name",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "<acctid>/*"
                    ]
                }
            }
        },
        {
            "Sid": "DynamoDBAccess",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem",
                "dynamodb:DeleteItem"
            ],
            "Resource": "arn:aws:dynamodb:<region>:<acctid>:table/terraform-state-lock-table-name"
        }
    ]
}

3. create a terraform backend file as in the example above with the role info
4. run terraform init

Additional Context

Yes, the assumable role has the StateBucketList statement with a prefix limitation.
This worked until version 1.5.7 but stoped on version 1.6.0

This is needed as the state bucket is shared and we do not want the ability to find all the accounts that are using the state bucket.

It seems to be related to a new request to the bucket (from the debug):

2023-11-11T23:19:44.828+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=ListObjectsV2 aws.region=<awsregion> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Workspaces tf_backend.req_id=4f1f7000-adcc-cc57-806e-5ec4fae58c68 tf_backend.s3.bucket=state-bucket-name tf_backend.workspace-prefix=env:/ http.url="https://state-bucket-name.s3.<awsregion>.amazonaws.com/?list-type=2&max-keys=1000&prefix=env%3A%2F" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.3 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.3 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.38.5" http.request.header.amz_sdk_invocation_id=<id> http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.method=GET net.peer.name=state-bucket-name.s3.<awsregion>.amazonaws.com http.request.header.authorization="AWS4-HMAC-SHA256 Credential=************/20231111/<awsregion>/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_security_token="*****" http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231111T211944Z http.request.body=""

See the tf_backend.workspace-prefix=env:/ which is being search in the root of the bucket for some reason, and that is not allowed by the statement.

References

No response

[padlyuck]

@simonweil I've got the same error. The reason for me was that my default AWS profile was configured to a different account than that which I used with Terraform. Here is my config:

terraform {
  backend "s3" {
    bucket         = "state-bucket-name"
    key            = "<AWS-ACCOUNT-ID>/<accountname>/<env>/terraform.tfstate"
    dynamodb_table = "terraform-state-lock-table-name"
    region         = "<theregion>"
+    access_key = "..." # same value as var.access_key
+    secret_key = "..." # same value as var.secret_key
  }
}
provider "aws" {
  region     = var.region
  access_key = var.access_key
  secret_key = var.secret_key
}

Another way I found to solve the issue is to add to ~/.aws/credentials another AWS profile with the creds you need for the terraform backend, and switch to this profile with export AWS_PROFILE=profile-name

[simonweil]

Thank you @padlyuck but this may be the same error message but not the same reason.
As noted in the description, the cause seems to be a regression in the workspace-prefix behavior and when I removed the restriction from the assumed role, it worked.

@jbardin any chance to get this resolved?

[crw]

@simonweil Just FYI @jbardin works on the core Terraform, whereas the AWS provider team at HashiCorp works on the S3 backend. Due to the recent changes to the S3 backend, the AWS provider team has been fairly responsive to new issues if they are related to the change, but it is ultimately up to their timeline and discretion to pick up these issues. Thanks for reporting this, and hopefully the team will get a chance to look at it.

[dinoDayo]

PSA for other people learning terraform - I got this same error but it was because the dynamo table and s3 bucket were not created yet. Once I manually created those resources via the console, my script worked as expected.

TL;DR: The resources referenced in the backend clause of your terraform script must exist before they can be referenced.

[simonweil]

@simonweil Just FYI @jbardin works on the core Terraform, whereas the AWS provider team at HashiCorp works on the S3 backend. Due to the recent changes to the S3 backend, the AWS provider team has been fairly responsive to new issues if they are related to the change, but it is ultimately up to their timeline and discretion to pick up these issues. Thanks for reporting this, and hopefully the team will get a chance to look at it.

Thank you for the clarification.

Any chance to get the attention of the AWS provider team at HashiCorp for this issue?

[crw]

@simonweil I did notify them of this issue, my understanding is that they are aware of it but I am not aware of whether it is currently prioritized. There were changes to the S3 backend in 1.6.6 (just released) but I do not see anything about it in the release notes :( -- normally they would close an issue (such as this) if it was fixed by a change.

[ewbankkit]

The AWS S3 ListObjectsV2 API is called when the S3 Backend is listing the workspaces in the configured bucket:

terraform/internal/backend/remote-state/s3/backend_state.go

Lines 45 to 56 in c7f052e

	params := &s3.ListObjectsV2Input{
	Bucket: aws.String(b.bucketName),
	Prefix: aws.String(prefix),
	MaxKeys: aws.Int32(maxKeys),
	}
	wss := []string{backend.DefaultStateName}
	ctx, baselog := baselogging.NewHcLogger(ctx, log)
	ctx = baselogging.RegisterLogger(ctx, baselog)
	pages := s3.NewListObjectsV2Paginator(b.s3Client, params)

Maybe the documentation is out-of-date WRT the required IAM permissions?
According to the AWS documentation, s3:ListBucket is the IAM permission required for ListObjectsV2.

[ewbankkit]

@simonweil Could you try setting the workspace_key_prefix to acctid, otherwise it looks like your StateBucketList permission Condition won't allow access to the "root" of the bucket.

[simonweil]

@simonweil Could you try setting the workspace_key_prefix to acctid, otherwise it looks like your StateBucketList permission Condition won't allow access to the "root" of the bucket.

Thank you for the pointer to this setting @ewbankkit
Tested using the workspace_key_prefix and it does work.
IMHO this is a workaround for a regression so I think it still needs to be fixed

[mmerickel]

This regression has prevented me from upgrading to 1.6/1.7 releases. We aren't using workspaces and if I specify a workspace_key_prefix in our backend config then terraform wants to migrate the state. On top of that, it can't migrate the state because to do that it's trying to do the ListObjectsV2 to sort out the migration, which also fails, leaving me stuck. So using 1.5.x until this is released.

In #34223 (comment) it's implied there were changes in 1.6.6 around the s3 backend. I tested 1.6.2, 1.6.3, 1.6.4, 1.6.5, and 1.6.6 and receive this ListObjectsV2 issue on all of them. 1.6.0 asks me to migrate state, so didn't go forward with trying that further.

Anyway looking forward to 1.6.7 to be released or #34513 to be merged into 1.7.1.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.