Bug 1776079: Azure: check existing role assignments before creating a new one #137

joelddiaz · 2019-11-25T20:01:11Z

Currently when re-reconciling an already-provisioned CredentialsRequest in Azure, the actuator will just always attempt to create a role assignment even if it already exists. There is error handling to catch the "RoleAssignmentExists" error, and it just moves along to the next task.

This results in the Resource Group where the cluster is installed having periodic entries in the Azure console Activity Log for the Resource Group recording these (non-critical) errors:

{
    "authorization": {
        "action": "Microsoft.Authorization/roleAssignments/write",
        "scope": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/jdiaz-az-gpqgx-rg/providers/Microsoft.Authorization/roleAssignments/94025186-5e7b-4e18-88de-4625cac3ed19"
    },
    "level": "Error",
    "operationName": {
        "value": "Microsoft.Authorization/roleAssignments/write",
        "localizedValue": "Create role assignment"
    },
    "resourceGroupName": "jdiaz-az-gpqgx-rg",
    "subStatus": {
        "value": "Conflict",
        "localizedValue": "Conflict (HTTP Status Code: 409)"
    },
    "properties": {
        "statusCode": "Conflict",
        "serviceRequestId": "931445cc-fbc6-469a-a6f4-2f7750788255",
        "statusMessage": "{\"error\":{\"code\":\"RoleAssignmentExists\",\"message\":\"The role assignment already exists.\"}}"
    },
}

Change the logic to pull the current list of Role Assignments so that we can avoid making unnecessary CreateRoleAssignment calls.

openshift-ci-robot · 2019-11-25T20:01:13Z

@joelddiaz: This pull request references Bugzilla bug 1776079, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1776079: Azure: check existing role assignments before creating a new one

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

joelddiaz · 2019-11-25T20:31:51Z

/bugzilla refresh

openshift-ci-robot · 2019-11-25T20:31:56Z

@joelddiaz: This pull request references Bugzilla bug 1776079, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dgoodwin

Were you able to reproduce what the customer found?

dgoodwin · 2019-11-26T12:53:18Z

pkg/azure/minter.go

+		// check whether assignment already exists
+		alreadyExists := false
+		for _, r := range currentRoleAssignments {
+			if strings.Contains(*r.Properties.Scope, scope) {


Can you add a comment here clarifying why we can't do an eq and what the data strings look like? I see scope above but what does r.Properties.Scope come out looking like? The test data looks awfully similar.

the Contains() comparison is because our generated scope var string doesn't have a leading /, and the returned r.Properties.Scope does have the leading /.

i reworked the generated string a few lines above to just put the leading / in the string so we can now do a simple string comparison. putting the leading / still allows all the minting to work.

joelddiaz · 2019-11-26T14:01:08Z

Were you able to reproduce what the customer found?

Yes. I stood up an Azure cluster and by forcing CCO to re-reconcile a CredentialsRequest, you could then see the log messages on the Azure console (that's how I got that JSON example ;) )

joelddiaz · 2019-11-26T16:02:52Z

/test e2e-azure

…new one Currently when re-reconciling an already-provisioned CredentialsRequest in Azure, the actuator will just always attempt to create a role assignment even if it already exists. There is error handling to catch the "RoleAssignmentExists" error, and it just moves along to the next task. This results in the Resource Group where the cluster is installed having periodic entries in the Activity Log recording these (non-critical) errors: ``` { "authorization": { "action": "Microsoft.Authorization/roleAssignments/write", "scope": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/jdiaz-az-gpqgx-rg/providers/Microsoft.Authorization/roleAssignments/94025186-5e7b-4e18-88de-4625cac3ed19" }, "level": "Error", "operationName": { "value": "Microsoft.Authorization/roleAssignments/write", "localizedValue": "Create role assignment" }, "resourceGroupName": "jdiaz-az-gpqgx-rg", "subStatus": { "value": "Conflict", "localizedValue": "Conflict (HTTP Status Code: 409)" }, "properties": { "statusCode": "Conflict", "serviceRequestId": "931445cc-fbc6-469a-a6f4-2f7750788255", "statusMessage": "{\"error\":{\"code\":\"RoleAssignmentExists\",\"message\":\"The role assignment already exists.\"}}" }, } ``` Change the logic to pull the current list of Role Assignments so that we can avoid making unnecessary CreateRoleAssignment calls.

joelddiaz · 2019-11-26T16:24:43Z

/test e2e-azure

dgoodwin · 2019-11-26T19:19:53Z

/lgtm

openshift-ci-robot · 2019-11-26T19:20:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, joelddiaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgoodwin,joelddiaz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-11-26T19:25:39Z

/retest