for_each fails when the number of resources should be known

[jsimoni]

Description

I am getting the following error message when the number of resources to create is known at plan

│ The "for_each" value depends on resource attributes that cannot be
│ determined until apply, so Terraform cannot predict how many instances will
│ be created. To work around this, use the -target argument to first apply
│ only the resources that the for_each depends on.

Terraform Version

- Terraform version:
Terraform v1.1.7
- Provider version(s):
registry.terraform.io/hashicorp/aws v4.4.0

Terraform Configuration Files

...

Debug Output

Expected Behavior

AWS SSO Group would be successfully assigned with the specified Accounts and Permission Sets

Actual Behavior

│ The "for_each" value depends on resource attributes that cannot be
│ determined until apply, so Terraform cannot predict how many instances will
│ be created. To work around this, use the -target argument to first apply
│ only the resources that the for_each depends on.

Steps to Reproduce

data "aws_organizations_organization" "this" {}

locals {
  account_id_list = data.aws_organizations_organization.this.accounts
}

module "permissionset" {
  source = "../../..//modules/permissionset"

  permission_set_name        = "name"
  permission_set_description = "description"
  awssso_instance_arn        = var.awsinstance_arn
  managed_policies_to_attach = ["arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"]
}

locals {
  account_permissionset_product = setproduct(var.account_id_list[*].id, var.permission_set_arn_list)
}

# associate the group to the permission set(s) & account(s)
resource "aws_ssoadmin_account_assignment" "group_permissionset_account_assignments" {
  for_each = {
    for association in local.account_permissionset_product : "${association[0]}-${association[1]}" => {
      account_id         = association[0]
      permission_set_arn = association[1]
    }
  }
  instance_arn       = var.awssso_instance_arn
  permission_set_arn = each.value.permission_set_arn

  principal_id   = data.aws_identitystore_group.awssso_group.group_id
  principal_type = "GROUP"

  target_id   = sensitive(each.value.account_id)
  target_type = "AWS_ACCOUNT"
}

Additional Context

If I use -target as suggested elsewhere (4149) to provision the permission set first, then the assignments works. However, the explanations I've seen for this error is that Terraform is unable to determine the number of resources to provision at plan so it fails. But in this case, while the arn of the permission set would be unknown, Terraform can determine the number of assignments to create.

References

[apparentlymart]

Hi @jsimoni,

Unfortunately what you observed here does seem to be Terraform working as designed, though I agree it's an annoying edge to the design.

The root problem here is that because map elements must have unique keys Terraform cannot know the number of elements in any map constructed with unique keys: Terraform has no way to prove that two of the keys won't end up having the same value.

Furthermore though, even ignoring the issue of number of elements for for_each in particular the map keys become part of the tracking address of each instance and so they must each individually be known during planning so Terraform can correlate with instances with the same keys during the apply phase.

So as commonly recommended in these situations, the answer is to set the map keys to include only known values and place unknown values such as a server-generated ID only in the values of the map elements. Terraform will then have the tracking keys it needs but can still take into account the unknown ID for use when populating arguments of the resource.

[jsimoni]

@apparentlymart is

set the map keys to include only known values and place unknown values such as a server-generated ID only in the values of the map elements.

documented anywhere? because it worked for me!

locals {
  account_permissionset_product = setproduct(var.account_id_list[*].id, var.permission_set_list)
}

# associate the group to the permission set(s) & account(s)
resource "aws_ssoadmin_account_assignment" "group_permissionset_account_assignments" {
  for_each = {
    for association in local.account_permissionset_product : "${association[0]}-${association[1].name}" => {
      account_id         = association[0]
      permission_set_arn = association[1].arn
    }
  }
  instance_arn       = var.awssso_instance_arn
  permission_set_arn = each.value.permission_set_arn

  principal_id   = data.aws_identitystore_group.awssso_group.group_id
  principal_type = "GROUP"

  target_id   = sensitive(each.value.account_id)
  target_type = "AWS_ACCOUNT"
}

[apparentlymart]

The intended documentation for this is Limitations on values in for_each, but I think we've learned in this issue that the docs there are not prescriptive enough about what to do about that.

We do have an updated error message for this case which aims to be more specific but I think that won't appear until the v1.2 release because it was a relatively recent change. Maybe we also updated the docs as part of that and so they too will appear with the v1.2 release, but if not it does seem like it would be helpful to spell out concretely the idea of using only known values in the key but being able to use unknown values in the value of each element.

[apparentlymart]

I just confirmed in #30327 that the new messaging includes a specific suggestion to make the keys be known during apply and put unknown values only in the value:

╷
│ Error: Invalid for_each argument
│ 
│   on for-each-unknown.tf line 10, in resource "null_resource" "other2":
│   10:   for_each = {
│   11:     (null_resource.example.id) = "baz"
│   12:   }
│     ├────────────────
│     │ null_resource.example.id is a string, known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot
│ be determined until apply, and so Terraform cannot determine the full set of
│ keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map
│ keys statically in your configuration and place apply-time results only in
│ the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only
│ the resources that the for_each value depends on, and then apply a second
│ time to fully converge.
╵

There's a limit to how specific we can be in error messages without making them super verbose but hopefully this new message coming in v1.2 would've been a better hook for learning how to move forward.

It looks like I did not update the documentation as part of that PR so there is still some potential for adding similar language to the documentation (where we can potentially elaborate more and show some examples) but at least the in-product messaging will be more accurate starting at the next minor release.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.