[Bug]: Assigning multiple role policies using aws_iam_role_policy_attachment fails with ConcurrentModification error

[jandryse]

Terraform Core Version

1.6.2

AWS Provider Version

5.25.0

Affected Resource(s)

aws_iam_role_policy_attachment

Expected Behavior

It should be possible to assign multiple policies to the same role.

Actual Behavior

When multiple policies are assigned, it very often fails with ConcurrentModification exception. It may be related to some changes in AWS, since the same code, with the same version of terraform and aws provider was working last week.

Relevant Error/Panic Output Snippet

Error: attaching policy arn:aws:iam::12545454:policy/test-policy-5 to IAM Role test-role: ConcurrentModification: Another request updating the entity is in progress. Please try again later.
│ 	status code: 409, request id: aaaaa-bbbbb-ccccc-ddddd-fea34e635b6a
│
│   with aws_iam_role_policy_attachment.test_attach[5],
│   on init.tf line 101, in resource "aws_iam_role_policy_attachment" "test_attach":
│  101: resource "aws_iam_role_policy_attachment" "test_attach" {

Terraform Configuration Files

variable "profile" {}
variable "region" {}

provider "aws" {
  profile = var.profile
  region  = var.region
}

terraform {
  required_version = ">= 1.0"
  backend "s3" {
    key = "spikes/role-benchmark/terraform.tfstate"
  }
  required_providers {
    aws = {
      source ="hashicorp/aws"
      version = "= 5.25.0"
    }

  }
}


data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "role" {
  name               = "test-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "aws_iam_policy_document" "policy" {
  statement {
    effect    = "Allow"
    actions   = ["ec2:Describe*"]
    resources = ["*"]
  }
}

resource "aws_iam_policy" "policy" {
  count = 8
  name        = "test-policy-${count.index}"
  description = "A test policy"
  policy      = data.aws_iam_policy_document.policy.json
}


resource "aws_iam_role_policy_attachment" "test_attach" {
  count = 8
  role       = aws_iam_role.role.name
  policy_arn = aws_iam_policy.policy[count.index].arn
}

Steps to Reproduce

apply the provided terraform. Or create new terraform, which assigns 8 polices to a role

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[flichtenheld]

I updated last week from 5.20.0 to 5.24.0 and started to see these errors. So I think this was introduced earlier than 5.25.0

[dezren39]

I switched
from (which pulled 5.25.0)

      version = "~>5.0"

to

      version = "5.24.0"

and apply now succeeds.

[flichtenheld]

This does not fail reliably for me. So it seems to depend on a race condition. Something to keep in mind when trying to bisect.

[flichtenheld]

This does not fail reliably for me. So it seems to depend on a race condition. Something to keep in mind when trying to bisect.

Sorry, to clarify: It does not fail reliably for me with our actual code. Tested the example given in the report and that fails always. Probably the high amount of attachments makes it easier to hit the race condition.

[rhowe]

I have started seeing this with version 3.76.1. I suspect AWS IAM behaviour has changed.

[jarpat]

Just wanted to report I am also seeing this with:

bash-4.2# terraform version
Terraform v1.4.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.4.0

as of today.

[flichtenheld]

I have started seeing this with version 3.76.1. I suspect AWS IAM behaviour has changed.

Yes, I tried bisecting this. But found no version with which it actually works. So it looks indeed like AWS-side change.

[flichtenheld]

terraform apply -parallelism=1 works around the problem.

[mgusiew-guide]

I haven't changed Terraform or Terraform AWS version recently and I started to see this today. I also observe the problem on older branches. So this is regression on AWS side. I agree that retries on conflict may be needed to solve this problem.

[venkatamutyala]

We had a regression suite run on our end early Friday without this issue. Saturday/Sunday have been failures. We run our regressions against us-west-2 but given IAM is a global service region specifics may not matter in this case. I also made some changes late yesterday to run against us-east-1 and was able to replicate the issue there to.

We are using 1.5.5 of terraform and the latest AWS provider. Downgrading a few minors didn't help at all. At the moment we just re-run the terraform and it's working.

[flichtenheld]

Tried to implement some testing for my patch. But now I can't reproduce the issue anymore. Maybe AWS fixed their stuff?

[venkatamutyala]

@flichtenheld thanks for making the PR. I just tried recreating the issue twice this evening (Pacific Time) and was unsuccessful. My team's infra-regression suites are working fine this afternoon, which is great. Hopefully, your PR can get merged in soon so that we can avoid this issue in the future.

[r0bnet]

We have the same issue but with aws_iam_user_policy_attachment and it started last week on Friday (10.11.2023). It's flaky and doesn't seem to appear all the times. It didn't happen for us anymore during the last 24 hours so maybe it's fixed on AWS side? Anyway thanks @flichtenheld for the PR.

[mgusiew-guide]

FTR the error went away yesterday afternoon so I contacted AWS to get some explanation. I was told that recently AWS made changes in IAM API, in particular AttachRolePolicy, AttachUserPolicy, and AttachGroupPolicy started to throw ConcurrentModificationException in case of concurrent requests. After the change AWS noticed elevated ConcurrentModification errors when calling IAM APIs. Because of that the issue has been resolved and the service is operating normally. Will try to clarify if "resolved" means that the feature was rolled back or something else.

[mgusiew-guide]

Got some updates from AWS side:

• The ConcurrentModificationException related change was rolled back
• Currently, it is not planned to roll out the changes in the future since it is impacting the customer's environment. There will be an announcement or notification regarding this if it is further planned to be implemented in the future

Hope that helps

[github-actions]

This functionality has been released in v5.26.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.