Inherited custom policy definitions are not available right after management group creation

[magodo]

I have a parent mgmt group, in which I have defined a custom policy definition. Then (after waiting for hours), I try to create a child mgmt group:

PUT https://management.azure.com/providers/Microsoft.Management/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe?api-version=2020-05-01 HTTP/2.0

Wait until the creation LRO finishes:

GET https://management.azure.com/providers/Microsoft.Management/operationResults/create/asyncOperation/status/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe?api-version=2020-05-01 HTTP/2.0

{
  "id": "/providers/Microsoft.Management/operationResults/create/asyncOperation/status/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe",
  "name": "7d97724c-c7fb-4b22-bbf8-c615fff88efe",
  "status": "Succeeded",
  "properties": {
    "tenantId": "a20e83fc-34d6-4c8e-8ae7-bf3d5eac71aa",
    "displayName": "ChildGroup",
    "details": {
      "version": 1,
      "updatedTime": "2022-03-11T06:47:22.8803754Z",
      "updatedBy": "e17e0665-5332-442b-910e-041e66f09a27"
    }
  }
}

Additionally, I'll try to GET this child mgmt group until it returns 200 (the first several calls will return 403):

GET https://management.azure.com/providers/Microsoft.Management/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe?%24expand=children&%24recurse=true&api-version=2020-05-01 HTTP/2.0

{
  "id": "/providers/Microsoft.Management/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe",
  "type": "Microsoft.Management/managementGroups",
  "name": "7d97724c-c7fb-4b22-bbf8-c615fff88efe",
  "properties": {
    "tenantId": "a20e83fc-34d6-4c8e-8ae7-bf3d5eac71aa",
    "displayName": "ChildGroup",
    "details": {
      "version": 0,
      "updatedTime": "0001-01-01T00:00:00",
      "updatedBy": null,
      "parent": {
        "id": "/providers/Microsoft.Management/managementGroups/7b7f297a-0331-4915-b8ec-4214505967b9",
        "name": "7b7f297a-0331-4915-b8ec-4214505967b9",
        "displayName": "ParentGroup"
      }
    },
    "children": null
  }
}

Right after this, I try to list the policy definitions with in this child mgmt group and expect that the custom policy definition I created above is inherited and included in the list result:

GET https://management.azure.com/providers/Microsoft.Management/managementGroups/7d97724c-c7fb-4b22-bbf8-c615fff88efe/providers/Microsoft.Authorization/policyDefinitions?api-version=2021-06-01 HTTP/2.0

(Also I'll follow the nextLink in the response until iteration done)

However, the custom policy definition can't be found in the list.

After waiting several minutes and try again, the custom policy definition is included in the list result.

In another try, I directly read the custom policy definition id and assign it to the child mgmt group right after it is created:

PUT https://management.azure.com//providers/Microsoft.Management/managementGroups/e0a965d1-b3e3-48ac-9b66-0793096f4a47/providers/Microsoft.Authorization/policyAssignments/example-policy?api-version=2021-06-01

But it returns:

{
  "error": {
    "code": "InvalidCreatePolicyAssignmentRequest",
    "message": "The policy definition specified in policy assignment example-policy is out of scope. Policy definitions should be specified only at or above the policy assignment scope. If the management groups hierarchy changed recently or if assigning a management group policy to new subscription, please allow up to 30 minutes for the hierarchy changes to apply and try again."
  }
}

Similarly, after waiting several minutes and try again, the assignment succeed.

I assume the reason might because there are sync issue between the policy RP and the mgmt group, which means the "creation success" return value from mgmt group doesn't necessarily mean all the inherited custom policy definitions are available in it.

From the client's point of view, is there something we can check to see when do the custom policy definitions are inherited in the child mgmt group? If the only solution for now is just to delay, then is there any official guidance on how long should we wait for?

[han-msazure]

Hi @magodo , this is primarily because of ARM's cache that keeps the mg information. As for solutions --

1. Wait for 30mins after creating the MG. ARM cache will be refreshed after that.
2. Log in/out. This can also refresh ARM's cache.

Please feel free to let me know if that works.

[magodo]

@han-msazure Thank you for the quick response! But what do you mean by log in/out if I'm calling the API directly from CLI?

[han-msazure]

Hi @magodo. Thank you for getting back. Could you try 'armclient login' or 'armclient clearcache' to refresh the token?
https://github.com/projectkudu/ARMClient

[magodo]

@han-msazure I'm using Azure SDK and ensure to get a refreshed token before sending the API. I don't think it has something to do with the token. Just want to be clear here is that what we want ultimately is a way to know when can we assign/query policy definitions for the mgmt group after it is created.

[han-msazure]

@magodo Please wait for 30mins after the MG is created.

[magodo]

@han-msazure Is there any official documentation that annouce this limitation, so that we can reference to?

[han-msazure]

@magodo We don't have a documentation for this scenario at the moment, but we would definitely improve it in the future.

[magodo]

@han-msazure Is there any work item/issue that can be tracked?

[han-msazure]

@magodo No there isn't.

[han-msazure]

Actually we do -- Manage your resources with management groups

Important
Azure Resource Manager user tokens and management group cache lasts for 30 minutes before they are forced to refresh. After doing any action like moving a management group or subscription, it might take up to 30 minutes to show. To see the updates sooner you need to update your token by refreshing the browser, signing in and out, or requesting a new token.

[magodo]

@han-msazure Thank you for the link! I've locally verified that refreshing the token works! BTW, do you mind share more insight on how this works behind the scenes? I mean there are other azure resources (e.g. keyvault) are suffering ARM caching issues like a LIST on the /resources API doesn't return the full reource list (see: Azure/AKS#1964 (comment)). I'm wondering whether refreshing a new token can solve that problem?

[han-msazure]

@magodo That would be a better question for key vault team to answer instead of policy or mg team.
As the token refresh works for you, i'm going to close this issue.