Summary

This project aims to provide a Terraform module to make it easy to setup Azure permissions needed to install and manage ARO. The problem we see in the field, is that there are overlapping identities that need specific permission sets and this often times results in incorrect permissions and makes for a confusing experience.

WARN: It should be noted that certain product changes will force this to change. This is a community-supported project and you should consult your appropriate product documentation prior to using this in your environment to ensure it is appropriate for your needs.

Identities

This section identifies the individual identities that need ARO permissions. These identities will be used in verbiage when describing what permissions are needed.

NOTE: the Install Flags column helps to associate these service principals to actual ARO installation flags using the CLI.

Identity	Type	Install Flags	Description
Cluster Service Principal	Service Principal	--client-id and --client-secret	Runs all operations in the cluster and that interacts with the Azure API as part of operations within the cluster. Cluster Autoscaler and the Service Controller for Azure are two key components which leverage these credentials.
Installer	User or Service Principal	N/A	Whomever the installation process is run as. This may be a user account or a service principal. This is who is logged in using the az login command.
Resource Provider Service Principal	Service Principal	N/A	Azure Resource Provider that represents ARO. This is automatically created in the account when first running the setup step az provider register -n Microsoft.RedHatOpenShift --wait. This service principal can be found by running az ad sp list --filter "displayname eq 'Azure Red Hat OpenShift RP'".

Objects

This section defines the objects which need individual permissions.

NOTE: the Install Flags column helps to associate these objects to actual ARO installation flags using the CLI.

Object Type	Install Flags	Description
Subscription	--subscription	The highest level a permission will be applied. Inherits down to all objects within that subscription. This is not a mandatory flag and the subscription may be set based on how a user has logged in with az login.
ARO Resource Group	--resource-group	Resource group in the above subscription where the actual ARO object is created.
Cluster Resource Group	--cluster-resource-group	Resource group in the above subscription where the underlying ARO objects (e.g. VMs, load balancers) are created. This is created automatically as part of provisioning.
Network Resource Group	--vnet-resource-group	Resource group in the above subscription where network resources (e.g. VNET, NSG) exist. Some organizations will use the Cluster Resource Group for this purpose as well and do not need a dedicated Network Resource Group.
VNET	--vnet	VNET where the ARO cluster will be provisioned.
Network Security Group	N/A	Only required for BYO-NSG scenarios. Network security group, applied to the subnets. This is is pre-applied by the user to the subnets prior to installation.
Disk Encryption Set	--disk-encryption-set	The disk encryption set used to encrypt master and worker node disks.

Permissions

This section identifies what permissions are needed by each individual identity.

NOTE: row numbers are used to indicate in the code where permissions are aligned.

Permission Number	Identity	Object	Permission	Comment
1	Cluster Service Principal	VNET	Network Contributor or Minimal Network Permissions
2	Cluster Service Principal	Network Security Group	Network Contributor or Minimal Network Permissions	Only needed if BYO-NSG is pre-attached to the subnet.
3	Cluster Service Principal	ARO Resource Group	Contributor
4	Cluster Service Principal	Disk Encryption Set	Other
5	Installer	ARO Resource Group	Contributor or Minimal ARO Permissions
6	Installer	Network Resource Group	Reader	Only required if az aro create is used to install.
7	Installer	Subscription	User Access Administrator	Only required if az aro create is used to install.
8	Installer	Azure AD	Directory Readers	Only required if az aro create is used to install.
9	Installer	VNET	Network Contributor or Minimal Network Permissions	Only required if az aro create is used to install.
10	Resource Provider Service Principal	VNET	Network Contributor or Minimal Network Permissions
11	Resource Provider Service Principal	Network Security Group	Network Contributor or Minimal Network Permissions
12	Resource Provider Service Principal	Disk Encryption Set	Other
13	Resource Provider Service Principal	Cluster Resource Group	Owner	This permission does not need to pre-exist. It is applied when the Resource Provider Service Principal creates the resource group as part of installation. This is for documentation purposes only.

Minimal Network Permissions

In many cases, such as separation of duties and where network teams must provide infrastructure to consume, a reduced permission set lower than Network Contributor is required. This is possible, however it should be noted that product documentation indicates higher permissions and the product will be developed against that assumption unless otherwise noted.

The following permission, in place of Network Contributor, have been successful (including links to the code which validates the permissions). The VNET this applies to equates to the value of the --vnet flag in the az aro create command:

Needed always:

• Microsoft.Network/virtualNetworks/join/action
• Microsoft.Network/virtualNetworks/read
• Microsoft.Network/virtualNetworks/write
• Microsoft.Network/virtualNetworks/subnets/join/action
• Microsoft.Network/virtualNetworks/subnets/read
• Microsoft.Network/virtualNetworks/subnets/write
• Microsoft.Network/networkSecurityGroups/join/action

Needed when provided VNET subnets have route table(s) attached:

• Microsoft.Network/routeTables/join/action
• Microsoft.Network/routeTables/read
• Microsoft.Network/routeTables/write

Needed when provided VNET subnets have NAT gateway(s) attached:

• Microsoft.Network/natGateways/join/action
• Microsoft.Network/natGateways/read
• Microsoft.Network/natGateways/write

Minimal ARO Permissions

In addition to minimizing network permissions, the installer role may need minimal permissions as well. These permissions are as follows:

• Microsoft.RedHatOpenShift/openShiftClusters/read
• Microsoft.RedHatOpenShift/openShiftClusters/write
• Microsoft.RedHatOpenShift/openShiftClusters/delete
• Microsoft.RedHatOpenShift/openShiftClusters/listCredentials/action
• Microsoft.RedHatOpenShift/openShiftClusters/listAdminCredentials/action

Other Permissions

In addition to the above, the following other permissions may be needed by specific identities:

• Microsoft.Compute/diskEncryptionSets/read

Prereqs

Prior to running this module, the following must be satisfied:

1. Must be logged in as an administrator user using the az login command. Because assigning permissions is an administrative task, it is assumed whomever is running this module is an administrator. Alternative to full tenant administrator permissions, a user that has the User Access Administrator role should be able to complete this task.

2. Must have the az CLI installed and configured locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now.

3. Must have the jq CLI installed locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now.

4. Must have a VNET architecture pre-deployed and used as an input.

Examples

Examples of how to use this module are located in the examples directory. The examples thus far are:

1. When ARO will be installed with the Azure az CLI - examples/cli
2. When ARO will be installed with an automation tool like Terraform (using the Microsoft API) - examples/api