azurerm_policy_set_definition policy_group_names not applying

[richardlock]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.0.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v2.81.0

Affected Resource(s)

• azurerm_policy_set_definition

Terraform Configuration Files

modules/azurerm_policy_set_definition/main.tf

terraform {
  experiments = [module_variable_optional_attrs]
}

resource "azurerm_policy_set_definition" "main" {
  description = var.description
  display_name = var.display_name
  management_group_name = var.management_group_name
  metadata = try(length(var.metadata) > 0, false) ? jsonencode(var.metadata) : null
  name = var.name
  parameters = try(length(var.parameters) > 0, false) ? jsonencode(var.parameters) : null
  policy_type = var.policy_type  

  dynamic "policy_definition_group" {
    for_each = var.policy_definition_groups

    content {
      additional_metadata_resource_id = policy_definition_group.value.additional_metadata_resource_id
      category = policy_definition_group.value.category
      description = policy_definition_group.value.description
      display_name = policy_definition_group.value.display_name
      name = policy_definition_group.value.name
    }
  }

  dynamic "policy_definition_reference" {
    for_each = var.policy_definition_references

    content {
      parameter_values = try(jsonencode(policy_definition_reference.value.parameter_values), null)
      policy_definition_id = policy_definition_reference.value.policy_definition_id
      policy_group_names = try(policy_definition_reference.value.policy_group_names, null)
      reference_id = try(policy_definition_reference.value.reference_id, null)
    }
  }
}

modules/azurerm_policy_set_definition/variables.tf

variable "description" {
  type = string
  default = null
  description = "Description of the policy set definition"
}

variable "display_name" {
  type = string
  description = "Display name of the policy set definition"
}

variable "management_group_name" {
  type = string
  default = null
  description = "Management group name of the policy set definition"
}

variable "metadata" {
  type = map
  default = null
  description = "Metadata for the policy set definition"
}

variable "name" {
  type = string
  description = "Name of the policy set definition"
}

variable "parameters" {
  type = string
  default = null
  description = "Parameters for the policy set definition (Json object)"
}

variable "policy_definition_groups" {
  type = list(object({
    additional_metadata_resource_id = optional(string)
    category = optional(string)
    description = optional(string)
    display_name = optional(string)
    name = string
  }))
  default = []
  description = "Policy definition group"
}

variable "policy_definition_references" {
  description = "List of policy definition references"
}

#variable "policy_definition_references" {
#  type = list(object({
#    parameter_values = optional(any)
#    policy_definition_id = string
#    policy_group_names = optional(list(string))
#    reference_id = optional(string)
#  }))
#  description = "Policy definition reference"
#}

variable "policy_type" {
  type = string
  default = "Custom"
  description = "Type of the policy set (Builtin or Custom)"
}

main.tf

locals {
  allowed_locations = ["UK South", "UK West"]
  required_tags = [
    "business-unit",
    "cost-centre",
    "environment",
    "location",
    "owner",
    "service",
    "technical-owner",
    "tier"
  ]
}

module "governance_test" {
  source = "./modules/azurerm_policy_set_definition"
  name = "governance-tst"
  description = "Governance Test policy initiative"
  display_name = "governance-tst"
  management_group_name = "test-mg"
  policy_type = "Custom"
  metadata = {
    category = "General"
  }
  policy_definition_groups = [
    {
      name = "Tags"
    }
  ]
  policy_definition_references = concat(
    [
      {
        parameter_values = {
          listOfAllowedLocations = {
            value = local.allowed_locations
          }
        }
        policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/e56962a6-4747-49cd-b67b-bf8b01975c4c"
        reference_id = "Allowed locations_1"
      },
      {
        parameter_values = {
          listOfAllowedLocations = {
            value = local.allowed_locations
          }
        }
        policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/e765b5de-1225-4ba3-bd56-1ac6695af988"
        reference_id = "Allowed locations for resource groups_1"
      },
    ],
    [
      for i, tag in local.required_tags : {
        parameter_values = {
          tagName = {
            value = tag
          }
        }
        policy_group_names = ["Tags"]
        policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/96670d01-0a4d-4649-9c89-2d3abc0a5025"
        reference_id = "Require a tag on resource groups_${i+1}"
      }
    ],
    [
      for i, tag in local.required_tags : {
        parameter_values = {
          tagName = {
            value = tag
          }
        }
        policy_group_names = ["Tags"]
        policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/ea3f2387-9b95-492a-a190-fcdc54f7b070"
        reference_id = "Inherit a tag from the resource group if missing_${i+1}"
      }
    ]
  )
}

Debug Output

N/A

Panic Output

N/A

Expected Behaviour

The policy initiative should have a group "Tags" created with policy definitions added to that group.

When querying the object using PowerShell after manually adding a policy definition to a group, the groupNames property shows the correct group.

(Get-AzPolicySetDefinition -Id /providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst).Properties.PolicyDefinitions | fl

policyDefinitionReferenceId : Allowed locations_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/e56962a6-4747-49cd-b67b-bf8b01975c4c
parameters                  : @{listOfAllowedLocations=}
groupNames                  : {}

policyDefinitionReferenceId : Allowed locations for resource groups_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/e765b5de-1225-4ba3-bd56-1ac6695af988
parameters                  : @{listOfAllowedLocations=}
groupNames                  : {}

policyDefinitionReferenceId : Require a tag on resource groups_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/96670d01-0a4d-4649-9c89-2d3abc0a5025
parameters                  : @{tagName=}
groupNames                  : {Tags}

Actual Behaviour

After running "terraform apply", the policy initiative is created and the group "Tags" is created, but no policy definitions are added to the group.

When querying the object using PowerShell, the groupNames property is missing from all policy definitions.

(Get-AzPolicySetDefinition -Id /providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst).Properties.PolicyDefinitions | fl

policyDefinitionReferenceId : Allowed locations_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/e56962a6-4747-49cd-b67b-bf8b01975c4c
parameters                  : @{listOfAllowedLocations=}

policyDefinitionReferenceId : Allowed locations for resource groups_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/e765b5de-1225-4ba3-bd56-1ac6695af988
parameters                  : @{listOfAllowedLocations=}

policyDefinitionReferenceId : Require a tag on resource groups_1
policyDefinitionId          : /providers/Microsoft.Authorization/policyDefinitions/96670d01-0a4d-4649-9c89-2d3abc0a5025
parameters                  : @{tagName=}

terraform apply
module.governance_test.azurerm_policy_set_definition.main: Refreshing state... [id=/providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.governance_test.azurerm_policy_set_definition.main has been changed
  ~ resource "azurerm_policy_set_definition" "main" {
        id                    = "/providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst"
        name                  = "governance-tst"
        # (7 unchanged attributes hidden)


      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              - "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
        # (4 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes,
the following plan may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.governance_test.azurerm_policy_set_definition.main will be updated in-place
  ~ resource "azurerm_policy_set_definition" "main" {
        id                    = "/providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst"
        name                  = "governance-tst"
        # (7 unchanged attributes hidden)


      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
      ~ policy_definition_reference {
          ~ policy_group_names   = [
              + "Tags",
            ]
            # (4 unchanged attributes hidden)
        }
        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│   on modules/azurerm_policy_set_definition/main.tf line 2, in terraform:
│    2:   experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.governance_test.azurerm_policy_set_definition.main: Modifying... [id=/providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst]
module.governance_test.azurerm_policy_set_definition.main: Modifications complete after 1s [id=/providers/Microsoft.Management/managementgroups/test-mg/providers/Microsoft.Authorization/policySetDefinitions/governance-tst]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Steps to Reproduce

1. terraform apply

Important Factoids

None.

References

N/A

[richardlock]

Hi. Any idea of timeframe to look at this one? Thanks.

[anwojcie]

Only on update

I can confirm the behavior at least with updates of a PolicySetDefinition.
It looks like the initial deployment is working as expected (group names are defined for Policy References)
This is valid for replacements as well tf apply -replace ....
But on update deployments this happens:

Pre

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.88.1"
    }
  }
}
provider "azurerm" {
  features {}
}

initial deployment with group names set

resource "azurerm_policy_set_definition" "PolicySet" {
  name                  = "GroupTestInitiative"
  policy_type           = "Custom"
  display_name          = "GroupTestInitiative"
  management_group_name = "est"

  policy_definition_group {
    name = "Group1"
  }
  policy_definition_group {
    name = "Group2"
  }

  policy_definition_reference {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb"
    policy_group_names   = ["Group1"]
  }
  policy_definition_reference {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a"
    policy_group_names   = ["Group2"]
  }

}

in trace log, groupnames are set on PUT

PUT /providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative?api-version=2019-09-01 HTTP/1.1
User-Agent: Go/go1.17.3 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 policy/2019-09-01 HashiCorp Terraform/1.0.4 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/2.88.1 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
{"properties":{"description":"","displayName":"GroupTestInitiative","policyDefinitionGroups":[{"name":"Group1"},{"name":"Group2"}],"policyDefinitions":[{"groupNames":["Group1"],"parameters":{},"policyDefinitionId":"/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb","policyDefinitionReferenceId":""},{"groupNames":["Group2"],"parameters":{},"policyDefinitionId":"/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a","policyDefinitionReferenceId":""}],"policyType":"Custom"}}: timestamp=2021-12-07T06:55:16.083+0100

Check via pwsh

❯ Get-AzPolicySetDefinition -Name GroupTestInitiative -ManagementGroupName est | ConvertTo-Json -Depth 99
{
  "Name": "GroupTestInitiative",
  "ResourceId": "/providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative",
  "ResourceName": "GroupTestInitiative",
  "ResourceType": "Microsoft.Authorization/policySetDefinitions",
  "SubscriptionId": null,
  "PolicySetDefinitionId": "/providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative",
  "Properties": {
    "Description": "",
    "DisplayName": "GroupTestInitiative",
    "Metadata": {
      "createdBy": "70000001-e00b-400f-b00b-300000000005",
      "createdOn": "2021-12-07T05:55:17.5843717Z",
      "updatedBy": null,
      "updatedOn": null
    },
    "Parameters": null,
    "PolicyDefinitionGroups": [
      {
        "name": "Group1"
      },
      {
        "name": "Group2"
      }
    ],
    "PolicyDefinitions": [
      {
        "policyDefinitionReferenceId": "5256048736259605031",
        "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb",
        "parameters": {},
        "groupNames": [
          "Group1"
        ]
      },
      {
        "policyDefinitionReferenceId": "2719031940633985590",
        "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a",
        "parameters": {},
        "groupNames": [
          "Group2"
        ]
      }
    ],
    "PolicyType": 1
  }
}

changed group name in tf

resource "azurerm_policy_set_definition" "PolicySet" {
  name                  = "GroupTestInitiative"
  policy_type           = "Custom"
  display_name          = "GroupTestInitiative"
  management_group_name = "est"

  policy_definition_group {
    name = "Group1"
  }
  policy_definition_group {
    name = "Group2"
  }

  policy_definition_reference {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb"
    policy_group_names   = ["Group1"]
  }
  policy_definition_reference {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a"
    policy_group_names   = ["Group1"]
  }

}

plan shows the expected changes.
in trace log, groupnames are set on GET (right before the PUT)
BUT in trace log, groupnames are NOT set on PUT

PUT /providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative?api-version=2019-09-01 HTTP/1.1
User-Agent: Go/go1.17.3 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 policy/2019-09-01 HashiCorp Terraform/1.0.4 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/2.88.1 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
{"properties":{"description":"","displayName":"GroupTestInitiative","metadata":{"createdBy":"70000001-e00b-400f-b00b-300000000005","createdOn":"2021-12-07T05:55:17.5843717Z","updatedBy":null,"updatedOn":null},"policyDefinitionGroups":[{"name":"Group1"},{"name":"Group2"}],"policyDefinitions":[{"parameters":{},"policyDefinitionId":"/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb","policyDefinitionReferenceId":"5256048736259605031"},{"parameters":{},"policyDefinitionId":"/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a","policyDefinitionReferenceId":"2719031940633985590"}],"policyType":"Custom"}}: timestamp=2021-12-07T07:05:19.210+0100

Check via pwsh

❯ Get-AzPolicySetDefinition -Name GroupTestInitiative -ManagementGroupName est | ConvertTo-Json -Depth 99
{
  "Name": "GroupTestInitiative",
  "ResourceId": "/providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative",
  "ResourceName": "GroupTestInitiative",
  "ResourceType": "Microsoft.Authorization/policySetDefinitions",
  "SubscriptionId": null,
  "PolicySetDefinitionId": "/providers/Microsoft.Management/managementgroups/est/providers/Microsoft.Authorization/policySetDefinitions/GroupTestInitiative",
  "Properties": {
    "Description": "",
    "DisplayName": "GroupTestInitiative",
    "Metadata": {
      "createdBy": "70000001-e00b-400f-b00b-300000000005",
      "createdOn": "2021-12-07T05:55:17.5843717Z",
      "updatedBy": "70000001-e00b-400f-b00b-300000000005",
      "updatedOn": "2021-12-07T06:05:20.6461404Z"
    },
    "Parameters": null,
    "PolicyDefinitionGroups": [
      {
        "name": "Group1"
      },
      {
        "name": "Group2"
      }
    ],
    "PolicyDefinitions": [
      {
        "policyDefinitionReferenceId": "5256048736259605031",
        "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/2d21331d-a4c2-4def-a9ad-ee4e1e023beb",
        "parameters": {}
      },
      {
        "policyDefinitionReferenceId": "2719031940633985590",
        "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/0a914e76-4921-4c19-b460-a2d36003525a",
        "parameters": {}
      }
    ],
    "PolicyType": 1
  }
}

[pascalkrielen]

I have the same issue, on initial deploy it is fine, after any adjustment it clears out the groups and keeps reporting the need to change it, running apply does not help, only a destroy and apply can get things in a correct state again.

[toby-p9]

Having the same issue, investigated with @aurelschwitter, he found out it seems to be a missing line in the terraform provider:

In the Update method, a line defining GroupNames is missing. It is present in the definition function just a couple of lines below.

Looks like a simple oversight and an easy fix. He'll go for a merge request, let's hope it gets approved.

[stephybun]

Closed by #19890

[github-actions]

This functionality has been released in v3.39.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.