Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Applying aws_s3_bucket_policy and aws_s3_bucket_public_access_block at the same time may cause an error #7628

Open
minamijoyo opened this issue Feb 21, 2019 · 4 comments

Comments

@minamijoyo
Copy link
Contributor

@minamijoyo minamijoyo commented Feb 21, 2019

Terraform Version

$ terraform -v
Terraform v0.11.11
+ provider.aws v1.59.0

Affected Resource(s)

  • aws_s3_bucket
  • aws_s3_bucket_policy
  • aws_s3_bucket_public_access_block

Terraform Configuration

provider "aws" {
  version = "= 1.59.0"
  region  = "us-east-1"
}

resource "aws_s3_bucket" "b" {
  bucket = "minamijoyo-public-access-block-test"
}

resource "aws_s3_bucket_policy" "b" {
  bucket = "${aws_s3_bucket.b.id}"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "MYBUCKETPOLICY",
  "Statement": [
    {
      "Sid": "IPAllow",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::minamijoyo-public-access-block-test/*",
      "Condition": {
         "IpAddress": {"aws:SourceIp": "127.0.0.1/32"}
      }
    }
  ]
}
POLICY
}

resource "aws_s3_bucket_public_access_block" "example" {
  bucket = "${aws_s3_bucket.b.id}"

  block_public_acls   = true
  block_public_policy = true
}

Debug Output

2392 2019-02-21T11:49:50.073+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: -----------------------------------------------------↲
2393 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: 2019/02/21 11:49:50 [DEBUG] [aws-sdk-go] DEBUG: Response s3/PutBucketPolicy Details:↲
2394 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: ---[ RESPONSE ]--------------------------------------↲
2395 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: HTTP/1.1 409 Conflict↲
2396 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: Connection: close↲
2397 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: Transfer-Encoding: chunked↲
2398 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: Content-Type: application/xml↲
2399 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: Date: Thu, 21 Feb 2019 02:49:49 GMT↲
2400 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: Server: AmazonS3↲
2401 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: X-Amz-Id-2: I2Fd71SFEnfx9m7SOjcCaF6G+ZdyDYMMk/3qzSk7ZhXZ9ERhAyVGzlKhtFYd3TRxwg5yHVVm+i0=↲
2402 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: X-Amz-Request-Id: B663B2CF1942B6E2↲
2403 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4:-↲
2404 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4:-↲
2405 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: -----------------------------------------------------↲
2406 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: 2019/02/21 11:49:50 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>↲
2407 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: <Error><Code>OperationAborted</Code><Message>A conflicting conditional operation is currently in progress against this resource. Please try again.</     Message><RequestId>B663B2CF1942B6E2</RequestId><HostId>I2Fd71SFEnfx9m7SOjcCaF6G+ZdyDYMMk/3qzSk7ZhXZ9ERhAyVGzlKhtFYd3TRxwg5yHVVm+i0=</HostId></Error>↲
2408 2019-02-21T11:49:50.920+0900 [DEBUG] plugin.terraform-provider-aws_v1.59.0_x4: 2019/02/21 11:49:50 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/PutBucketPolicy failed, not retrying, error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.↲

Panic Output

No panic

Expected Behavior

No error

Actual Behavior

Got an error

$ terraform apply

Error: Error applying plan:

1 error(s) occurred:

* aws_s3_bucket_policy.b: 1 error(s) occurred:

* aws_s3_bucket_policy.b: Error putting S3 policy: OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
        status code: 409, request id: B663B2CF1942B6E2, host id: I2Fd71SFEnfx9m7SOjcCaF6G+ZdyDYMMk/3qzSk7ZhXZ9ERhAyVGzlKhtFYd3TRxwg5yHVVm+i0=

Steps to Reproduce

  1. terraform apply.

Important Factoids

Success and failure depend on timing.
I tried it a couple of times, but in my environment, there are fewer cases without the error.

References

Although the types of resources are different, calling S3 API in parallel to the same bucket may cause this error.

@bflad bflad added the service/s3 label Feb 21, 2019
@conn

This comment has been minimized.

Copy link

@conn conn commented Mar 5, 2019

I am also having issues with this bug:

resource "aws_s3_bucket" "this" {
  bucket_prefix = "xxxx"
  acl           = "private"

  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket = "${aws_s3_bucket.this.bucket}"

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_role" "policy_identifiers" {
  name = "xxxx"
}

data "aws_iam_policy_document" "s3_bucket_policy_policy" {
  version = "2012-10-17"

  statement {
    effect  = "Allow"
    actions = ["s3:*"]

    resources = [
      "${aws_s3_bucket.this.arn}/*",
      "${aws_s3_bucket.this.arn}",
    ]

    principals {
      type        = "AWS"
      identifiers = ["${data.aws_iam_role.policy_identifiers.arn}"]
    }
  }
}

resource "aws_s3_bucket_policy" "this" {
  bucket = "${aws_s3_bucket.this.bucket}"
  policy = "${data.aws_iam_policy_document.s3_bucket_policy_policy.json}"
}
Error: Error applying plan:

1 error(s) occurred:

* aws_s3_bucket_policy.this: 1 error(s) occurred:

* aws_s3_bucket_policy.this: Error putting S3 policy: OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
	status code: 409, request id: xxxxxxxxxxxxxxxx, host id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
@conn

This comment has been minimized.

Copy link

@conn conn commented Mar 5, 2019

Using depends_on is a good way to force the public access block and policy to be applied one-by-one instead of concurrently:

resource "aws_s3_bucket_policy" "this" {
  depends_on = ["aws_s3_bucket_public_access_block.this"]

This worked for me.

@minamijoyo

This comment has been minimized.

Copy link
Contributor Author

@minamijoyo minamijoyo commented Mar 6, 2019

I think that depends_on would work as a workaround, but it does not actually depend on it, we should implement an appropriate error handling. Since similar problems can occur with many s3 related resources, I'm not sure how do we handle it is the best.

@conn

This comment has been minimized.

Copy link

@conn conn commented Mar 6, 2019

Either that or sort out planned API calls to S3 so that they happen in sequence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.