Client refresh / credential caching issues in dev and prod

[jim-minter]

Saw an az aro create fail with the following storage deployment error messages:

{
    "error": {
        "code": "AuthorizationFailed",
        "message": "The client 'b9bc1545-a684-4e50-9a4d-2e2ee90bed6a' with object id 'b9bc1545-a684-4e50-9a4d-2e2ee90bed6a' does not have authorization to perform action 'Microsoft.Storage/storageAccounts/write' over scope '/subscriptions/225e02bc-43d0-43d1-a01a-17e584a4ef69/resourcegroups/aro-6hwuovvq/providers/Microsoft.Storage/storageAccounts/clusterixgbq' or the scope is invalid. If access was recently granted, please refresh your credentials."
    }
}

Need to understand how to fix this.

[jim-minter]

@julienstroheker we sometimes see this error, both in dev and in prod. It is like https://docs.microsoft.com/en-us/azure/role-based-access-control/troubleshooting#rbac-changes-are-not-being-detected . I assume there is some caching somewhere in ARM. I understand we should make the RBAC change, then refresh our access token; the problem is I think we are doing that correctly and we still get the error occasionally. Please can you look into this and try to understand what else we need to do? Does the v3 RP do additional tricks here?

[asalkeld]

I also get this too

ERRO[2020-02-21T12:01:30+10:00] pkg/frontend/frontend.go:268 frontend.reply() authorization.PermissionsClient#ListForResource: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '99d7cb4b-e087-4c94-8ced-3e15b07a870a' with object id '99d7cb4b-e087-4c94-8ced-3e15b07a870a' does not have authorization to perform action 'Microsoft.Authorization/permissions/read' over scope '/subscriptions/225e02bc-43d0-43d1-a01a-17e584a4ef69/resourcegroups/v4-australiasoutheast/providers/Microsoft.Network/virtualNetworks/dev-vnet/providers/Microsoft.Authorization' or the scope is invalid. If access was recently granted, please refresh your credentials."  component=frontend correlation-id= request-id=9bac8f35-12df-4aaa-8f9f-74f70c075979

[jim-minter]

https://gist.github.com/mjudeikis/81dc0f875d794992fb724df1f0c0ada8

[jim-minter]

dynamic validation has moved to backend, closing this.