Error destroying aws_ssoadmin resources

[jsimoni]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.4.0

Affected Resource(s)

• aws_ssoadmin_permission_set
• aws_ssoadmin_account_assignment
• aws_ssoadmin_managed_policy_attachment

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

locals {
  okta_awsinstance_arn    = "REDACTED"
  okta_identity_store_id  = "REDACTED"
}

data "aws_organizations_organization" "this" {}

data "aws_identitystore_group" "operations_group" {
  identity_store_id = local.okta_identity_store_id

  filter {
    attribute_path  = "DisplayName"
    attribute_value = "Team"
  }
}

resource "aws_ssoadmin_permission_set" "operations" {
  name             = "operations"
  description      = "operations"
  instance_arn     = local.okta_awsinstance_arn
  session_duration = "PT2H"
}

resource "aws_ssoadmin_managed_policy_attachment" "operations" {
  for_each = toset( ["arn:aws:iam::aws:policy/AWSCloudShellFullAccess", "arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"] )
  instance_arn       = local.okta_awsinstance_arn
  managed_policy_arn = each.value
  permission_set_arn = aws_ssoadmin_permission_set.operations.arn
}

resource "aws_ssoadmin_account_assignment" "operations" {
  for_each           = toset(data.aws_organizations_organization.this.accounts[*].id)
  instance_arn       = local.okta_awsinstance_arn
  permission_set_arn = aws_ssoadmin_permission_set.operations.arn

  principal_id   = data.aws_identitystore_group.operations_group.group_id
  principal_type = "GROUP"

  target_id   =  sensitive(each.value)
  target_type = "AWS_ACCOUNT"
}

Debug Output

Plan: 0 to add, 0 to change, 6 to destroy.
aws_ssoadmin_managed_policy_attachment.operations["arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"]: Destroying... 
aws_ssoadmin_account_assignment.operations["AccountA"]: Destroying... 
aws_ssoadmin_managed_policy_attachment.operations["arn:aws:iam::aws:policy/AWSCloudShellFullAccess"]: Destroying... 
aws_ssoadmin_account_assignment.operations["AccountB"]: Destroying... 
aws_ssoadmin_account_assignment.operations["AccountC"]: Destroying...
aws_ssoadmin_account_assignment.operations["AccountA"]: Destruction complete after 5s
aws_ssoadmin_account_assignment.operations["AccountB"]: Destruction complete after 5s
2022-03-08T06:25:31.253-0500 [ERROR] vertex "aws_ssoadmin_managed_policy_attachment.operations[\"arn:aws:iam::aws:policy/AWSCloudShellFullAccess\"] (destroy)" error: error waiting for SSO Permission Set (REDACTED) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)
aws_ssoadmin_account_assignment.operations["AccountC"]: Destruction complete after 5s
2022-03-08T06:25:33.075-0500 [ERROR] vertex "aws_ssoadmin_managed_policy_attachment.operations[\"arn:aws:iam::aws:policy/job-function/ViewOnlyAccess\"] (destroy)" error: error waiting for SSO Permission Set (REDACTED) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

│ Error: error waiting for SSO Permission Set (arn:aws:sso:::REDACTED) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)
│
│ Error: error waiting for SSO Permission Set (arn:aws:sso:::REDACTED) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

Panic Output

Expected Behavior

The AWS resources should be destroyed without Terraform errors.

Actual Behavior

Terraform generates errors on the first attempt to destroy. If I run destroy again immediately, it succeeds. It's not a race condition as I've waited hours between running apply & destroy. It may be that the dependencies aren't being tracked correctly since the second attempt to destroy always seems to succeed.

Steps to Reproduce

1. terraform apply
2. terraform destroy

Important Factoids

References

• #0000

[gdavison]

Thanks for submitting this, @jsimoni. Can you send the debug output, as described at https://www.terraform.io/internals/debugging, please?

[jsimoni]

@gdavison here is the debug output... https://github.com/jsimoni/23585/blob/main/debug-destroy.log

[jsimoni]

@gdavison do you have any thoughts on this one? Do you need more info from me?

[nickgautier]

I also encouter this issue when attempting account assignment.

  instance_arn       = "arn:aws:sso:::instance/ssoins-xxxxxxx"
  permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-xxxxxxxx/ps-xxxxxxxxx"
  # instance_arn       = data.aws_ssoadmin_permission_set.example.instance_arn
  # permission_set_arn = data.aws_ssoadmin_permission_set.example.arn

  principal_id   = data.aws_identitystore_group.example.group_id
  principal_type = "GROUP"

  target_id          = "xxxxxxx"
  target_type = "AWS_ACCOUNT"
}```
Output:
```╷
│ Error: error waiting for SSO Account Assignment for GROUP (xxxxx-30c1-7045-affe-1b2fc5de4e6a) to be created: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)
│ 
│   with aws_ssoadmin_account_assignment.example,
│   on main.tf line 46, in resource "aws_ssoadmin_account_assignment" "example":
│   46: resource "aws_ssoadmin_account_assignment" "example" {
│ 
╵```
Keen on any ideas to resolve looks to me like the API call to AWS itself might be failing.

[psilohanf]

For me destroying the account assignment worked but it failed removing Permission Sets:

╷
│ Error: error waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)
│
│
╵

[popsicleslayer]

Encountered the same issue while attaching Customer Managed policies to a Permission Set:

Error: error waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s()
│
│ with module.example_permission_set.aws_ssoadmin_customer_managed_policy_attachment.this[example-customer-managed-policy"],
│ on .terraform/modules/readonly_permission_set/main.tf line 36, in resource "aws_ssoadmin_customer_managed_policy_attachment" "this":
│ 36: resource "aws_ssoadmin_customer_managed_policy_attachment" "this" {
│

The customer managed policy gets attached to the permission set in the second attempt, but both times it failed with this error.

[jeremymturner]

I'm getting the same thing. In absence of a fix, is there a way to work around it? Is it just a matter of deleting the resources and terraform state objects manually and then trying again?

[bhanukolli]

We are also facing the same issue. Any workaround?

[absa-rsuarez]

Also seeing this issue but only for one of the permission sets customer managed policy attachments. The others were fine.

Error: error waiting for SSO Permission Set (redacted) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

[akshaytakalkar-ops]

Facing same error, from last 3months and it not addressed in new version of aws providers

Error: error waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ss-xx) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s()

[andremorais-sigtech]

Having the same issue while creating aws_ssoadmin_customer_managed_policy_attachment

error waiting for SSO Account Assignment for GROUP (xxxxx) to be created: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

terraform = "v1.2.5"
registry.terraform.io/hashicorp/aws = "4.50.0"

Any updates on this issue?

[andremorais-sigtech]

As a workaround, you can use aws_ssoadmin_permission_set_inline_policy instead, it does not trigger this issue.

[ls-pieter-vanisacker]

We also ran into this error, when enabling terraform DEBUG logging it was shown that we actually ran into an AWS IAM limit.
After increasing these limits the problem was solved and TF completed successfully.

[andremorais-sigtech]

While attaching a customer managed policy
TF Debug:

FailureReason":"Received a 404 status error: Not supported policy arn:aws:iam::xxxxxx:policy/xxxxxx.","PermissionSetArn":"arn:aws:sso:::permissionSet/ssoins-xxxxx/ps-xxxx","PrincipalId":"xxxxxx","PrincipalType":"GROUP","RequestId":"3d7afdbf-9fb3-4662-ba15-a44135549d30","Status":"FAILED","TargetId":"xxxxxxxx","TargetType":"AWS_ACCOUNT"}}" 

Although, the policy gets created and you can manually attach it. Also creating an inline policy works

[chris-stetter]

We also ran into this error, when enabling terraform DEBUG logging it was shown that we actually ran into an AWS IAM limit. After increasing these limits the problem was solved and TF completed successfully.

@ls-pieter-vanisacker Can you share which limits you have increased and how? Throughh ttps://console.aws.amazon.com/servicequotas/home?

[ls-pieter-vanisacker]

@ls-pieter-vanisacker in our case it was the amount of policies/roles that can be attached to a user/group. It defaults to 10, we had to increase it to 20.
Indeed through the service quotas UI. Crucial thing: you have to select the us-east-1 region for increasing IAM limits.

[rymancl]

We get the same error on every destroy of our sso-admin resources.

 Error: provisioning SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-<redacted>/ps-<redacted>): error waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-<redacted>/ps-<redacted>) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

Rerunning the destroy immediately will show the destroy plan for the aws_ssoadmin_permission_set resources that didn't get deleted before. The destroy will work fine on this second shot.

Pretty annoying though...

[tachomi]

Any update on this? I'm getting the same issue. Re executing the destroy command will work, however for automation purposes, this is not a workaround

Error: provisioning SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-XXXXXXX/ps-XXXXXXX): error waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-XXXXXXX/ps-XXXXXXX) to provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

[mocknen]

For those who are having trouble with this issue, try allowing the following actions for the resource arn:aws:iam::${Account}:role/aws-reserved/sso.amazonaws.com/${Region}/AWSReservedSSO_* to users running terraform.

• iam:GetRole
• iam:CreateRole
• iam:DeleteRole
• iam:ListAttachedRolePolicies
• iam:ListRolePolicies
• iam:AttachRolePolicy
• iam:DetachRolePolicy

Apparently, the action ProvisionPermissionSet calls the AWS internal action UpdateApplicationProfileForAWSAccountInstance, which is not documented, requiring the above permissions.

[mikeplem]

I am not sure if this adds to the conversation but I was getting a very similar error and when I looked at Cloudtrail logs for the UpdateApplicationProfileForAWSAccountInstance event name, I was getting an Access Denied error because the action was trying to run sso:ProvisionPermissionSet against the account the permission set was applied.

My Terraform is not associating any permission set to an account, it only creates the permission sets.

"errorMessage": "User: arn:aws:sts::ACCOUNT_ID_A:assumed-role/permission-set-deploy/awspermissionset-runner is not authorized to perform: sso:ProvisionPermissionSet on resource: arn:aws:sso:::account/ACCOUNT_ID_B because no identity-based policy allows the sso:ProvisionPermissionSet action"

[novekm]

Same issue here - wondering is it's something around how the state is handled for the resources during deletion? Running TF in debug mode gives me this information:

"FailureReason":"Received a 404 status error: Assignment not found."
"FailureReason":"Received a 404 status error: Permission set provision not found in AWS account 11111111111"

So it looks like it is unaware the resource is already deleted until you re-run terraform destroy for some reason.

[samlangley1]

This is a nice change however I don't believe it solves the issue raised here. The permission set still fails to be attached until the 2nd attempt.

[github-actions]

This functionality has been released in v5.14.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[novekm]

I am still receiving the same error upon running terraform destroy: Error: waiting for SSO Permission Set (arn:aws:sso:::permissionSet/ssoins-xxx/ps-xxx) provision: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: Received a 404 status error: Permission set provision not found in AWS account 123456789012. It is odd that a deletion is showing as a provision (which should align with a create operation).

It appears as if it is still not aware that the resource has already been deleted until you re-run terraform destroy again. The addition of the timeouts block doesn't seem to solve this (the error message appears within 60sec in my testing, well within the listed 5-10min default timeouts that were added). Perhaps there needs to be a retry, or the order of deletion needs to be modified?

[gdavison]

@samlangley1 and @novekm can you please create new issues. We typically do not check comments on closed issues, and I only came across this incidentally.

@novekm, in your particular case, the provision message is output when updating the resource. Can you include the Terraform configuration used?

[novekm]

Thanks @gdavison - here is the link to the new GitHub issue for this. Happy to try to implement the fix, I think I have an idea what it could be. Added that to the issue as well

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.