OpenShift Container Platform Deployment Template for Azure Stack

Bookmark aka.ms/OpenShift for future reference.

For OpenShift Origin refer to https://github.com/Microsoft/openshift-origin

OpenShift Container Platform 3.9 with Username / Password authentication for OpenShift in Azure Stack

This template deploys OpenShift Container Platform in Azure Stack with basic username / password for authentication to OpenShift. It includes the following resources:

Resource	Properties
Virtual Network	Address prefix: 10.0.0.0/8 Master subnet: 10.1.0.0/16 Node subnet: 10.2.0.0/16
Master Load Balancer	1 probes and 1 rule for TCP 443 NAT rules for SSH on Ports 2200-220X
Infra Load Balancer	2 probes and 2 rules for TCP 80 and TCP 443
Public IP Addresses	Bastion Public IP for Bastion Node OpenShift Master public IP attached to Master Load Balancer OpenShift Router public IP attached to Infra Load Balancer
Storage Accounts Unmanaged Disks	1 Storage Account for Bastion VM 1 Storage Account for Master VMs 1 Storage Account for Infra VMs 2 Storage Accounts for Node VMs 2 Storage Accounts for Diagnostics Logs 1 Storage Account for Private Docker Registry 1 Storage Account for Persistent Volumes
Network Security Groups	1 Network Security Group for Bastion VM 1 Network Security Group Master VMs 1 Network Security Group for Infra VMs 1 Network Security Group for Node VMs
Availability Sets	1 Availability Set for Master VMs 1 Availability Set for Infra VMs 1 Availability Set for Node VMs
Virtual Machines	1 Bastion Node - Used to Run Ansible Playbook for OpenShift deployment 3 or 5 Master Nodes 2 or 3 Infra Nodes User-defined number of Nodes (1 to 30) All VMs include a single attached data disk for Docker thin pool logical volume

READ the instructions in its entirety before deploying!

This template is referencing a RHEL image that will most likely not be valid for you. In order to deploy OpenShift to your Azure Stack, please fork this repo and update azuredeploy.json for the imageReference variable at line 378 to reference your image.

Currently, the Azure Cloud Provider does not work in Azure Stack. This means you will not be able to use disk attach for persistent storage in Azure Stack. You can always configure other storage options such as NFS, iSCSI, Gluster, etc. that can be used for persistent storage. We are exploring options to address the Azure Cloud Provider in Azure Stack but this will take a little bit of time.

As an alternative, you can choose to enable CNS and use Gluster for persistent storage. If CNS is enabled, three additional nodes will be deployed with additional storage for Gluster.

Additional documentation for deploying OpenShift in Azure can be found here: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/openshift-get-started

This template deploys multiple VMs and requires some pre-work before you can successfully deploy the OpenShift Cluster. If you don't get the pre-work done correctly, you will most likely fail to deploy the cluster using this template. Please read the instructions completely before you proceed.

After successful deployment, the Bastion Node is no longer required unless you want to use it to add nodes or run other playbooks in the future. You can turn it off and delete it or keep it around for running future playbooks. You can also use this as the jump host for managing your OpenShift cluster.

Prerequisites

Configure Azure CLI to connect to your Azure Stack environment

In order to use the Azure CLI to manage resources in your Azure Stack environment, you will need to first configure the CLI to connect to your Azure Stack. Follow instructions here: https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-connect-cli.

You will need to determine the necessary endpoints for each of the following settings:

• endpoint-resource-manager
• suffix-storage-endpoint
• suffix-keyvault-dns
• endpoint-active-directory-graph-resource-id
• endpoint-vm-image-alias-doc

Generate SSH Keys

You'll need to generate an SSH key pair (Public / Private) in order to provision this template. Ensure that you do NOT include a passphrase with the private key.

If you are using a Windows computer, you can download puttygen.exe. You will need to export to OpenSSH (from Conversions menu) to get a valid Private Key for use in the Template.

From a Linux or Mac, you can just use the ssh-keygen command. Once you are finished deploying the cluster, you can always generate new keys that uses a passphrase and replace the original ones used during initial deployment.

Create Key Vault to store SSH Private Key

You will need to create a Key Vault to store your SSH Private Key that will then be used as part of the deployment. This extra work is to provide security around the Private Key - especially since it does not have a passphrase. I recommend creating a Resource Group specifically to store the KeyVault. This way, you can reuse the KeyVault for other deployments and you won't have to create this every time you chose to deploy another OpenShift cluster.

1. Create Key Vault using Azure CLI 2.0
   a. Create new Resource Group: az group create -n <name> -l <location>
   Ex: az group create -n ResourceGroupName -l 'East US'
   b. Create Key Vault: az keyvault create -n <vault-name> -g <resource-group> -l <location> --enabled-for-template-deployment true
   Ex: az keyvault create -n KeyVaultName -g ResourceGroupName -l 'East US' --enabled-for-template-deployment true
   c. Create Secret: az keyvault secret set --vault-name <vault-name> -n <secret-name> --file <private-key-file-name>
   Ex: az keyvault secret set --vault-name KeyVaultName -n SecretName --file ~/.ssh/id_rsa
   

Red Hat Subscription Access

For security reasons, the method for registering the RHEL system has been changed to allow the use of an Organization ID and Activation Key as well as a Username and Password. Please know that it is more secure to use the Organization ID and Activation Key.

You can determine your Organization ID by running subscription-manager identity on a registered machine. To create or find your Activation Key, please go here: https://access.redhat.com/management/activation_keys.

You will also need to get the Pool ID that contains your entitlements for OpenShift. You can retrieve this from the Red Hat portal by examining the details of the subscription that has the OpenShift entitlements. Or you can contact your Red Hat administrator to help you.

azuredeploy.Parameters.json File Explained

1. _artifactsLocation: URL for artifacts (json, scripts, etc.)
2. masterVmSize: Size of the Master VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file
3. infraVmSize: Size of the Infra VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file
4. nodeVmSize: Size of the App Node VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file
5. storageKind: The type of storage to be used. Value can only be "unmanaged". When Azure Stack supports managed disk, this will be updated
6. openshiftClusterPrefix: Cluster Prefix used to configure hostnames for all nodes - bastion, master, infra and app nodes. Between 1 and 20 characters
7. masterInstanceCount: Number of Masters nodes to deploy
8. infraInstanceCount: Number of infra nodes to deploy
9. nodeInstanceCount: Number of Nodes to deploy
10. dataDiskSize: Size of data disk to attach to nodes for Docker volume - valid sizes are 32 GB, 64 GB, 128 GB, 256 GB, 512 GB, 1024 GB, and 2048 GB
11. adminUsername: Admin username for both OS (VM) login and initial OpenShift user
12. openshiftPassword: Password for OpenShift user
13. enableMetrics: Enable Metrics - value is either "true" or "false"
14. enableLogging: Enable Logging - value is either "true" or "false"
15. enableCNS: Enable Container Native Storage (CNS) - value is either "true" or "false". If set to true, 3 additional nodes will be deployed for CNS
16. rhsmUsernameOrOrgId: Red Hat Subscription Manager Username or Organization ID. To find your Organization ID, run on registered server: subscription-manager identity.
17. rhsmPasswordOrActivationKey: Red Hat Subscription Manager Password or Activation Key for your Cloud Access subscription. You can get this from here.
18. rhsmPoolId: The Red Hat Subscription Manager Pool ID that contains your OpenShift entitlements
19. sshPublicKey: Copy your SSH Public Key here
20. keyVaultResourceGroup: The name of the Resource Group that contains the Key Vault
21. keyVaultName: The name of the Key Vault you created
22. keyVaultSecret: The Secret Name you used when creating the Secret (that contains the Private Key)
23. enableAzure: Enable Azure Cloud Provider - value is only "false". When Azure Cloud Provider is supported in Azure Stack, this will be updated
24. aadClientId: Currently not used. Place holder for when Azure Cloud Provider is supported in Azure Stack. Azure Active Directory Client ID also known as Application ID for Service Principal
25. aadClientSecret: Currently not used. Place holder for when Azure Cloud Provider is supported in Azure Stack. Azure Active Directory Client Secret for Service Principal
26. defaultSubDomainType: This will either be nipio (if you don't have your own domain) or custom if you have your own domain that you would like to use for routing
27. defaultSubDomain: The wildcard DNS name you would like to use for routing if you selected custom above. If you selected nipio above, you must still enter something here but it will not be used

Deploy Template

Once you have collected all of the prerequisites for the template, you can deploy the template by clicking Deploy to Azure or populating the azuredeploy.parameters.json file and executing Resource Manager deployment commands with PowerShell or the Azure CLI.

Azure CLI 2.0

1. Create Resource Group: az group create -n <name> -l <location>
   Ex: az group create -n openshift-cluster -l westus
2. Create Resource Group Deployment: az group deployment create --name <deployment name> --template-file <template_file> --parameters @<parameters_file> --resource-group <resource group name> --nowait
   Ex: az group deployment create --name ocpdeployment --template-file azuredeploy.json --parameters @azuredeploy.parameters.json --resource-group openshift-cluster --no-wait

NOTE

The Service Catalog, Ansible Service Broker, and Template Service Broker will only deploy if CNS is enabled due to requirement for persistent storage.

Be sure to follow the OpenShift instructions to create the necessary DNS entry for the OpenShift Router for access to applications.

TROUBLESHOOTING

If you encounter an error during deployment of the cluster, please view the deployment status. The following Error Codes will help to narrow things down.

1. Exit Code 3: Your Red Hat Subscription User Name / Password or Organization ID / Activation Key is incorrect
2. Exit Code 4: Your Red Hat Pool ID is incorrect or there are no entitlements available
3. Exit Code 5: Unable to provision Docker Thin Pool Volume
4. Exit Code 6: OpenShift Cluster installation failed
5. Exit Code 11: Metrics failed to deploy
6. Exit Code 12: Logging failed to deploy

For further troubleshooting, please SSH into your Bastion node on port 22. You will need to be root (sudo su -) and then navigate to the following directory: /var/lib/waagent/custom-script/download

You should see a folder named '0' and '1'. In each of these folders, you will see two files, stderr and stdout. You can look through these files to determine where the failure occurred.

Post-Deployment Operations

Metrics and logging

Metrics

If you deployed Metrics, it will take a few extra minutes deployment to complete. Please be patient. If enableCNS is set to true, then Metrics will use Gluster for persistent storage. Otherwise, it will use ephemeral storage.

Once the deployment is complete, log into the OpenShift Web Console and complete an addition configuration step. Go to the openshift-infra project, click on Hawkster metrics route, and accept the SSL exception in your browser.

Logging

If you deployed Logging, it will take a few extra minutes deployment to complete. Please be patient. If enableCNS is set to true, then Logging will use Gluster for persistent storage. Otherwise, it will use ephemeral storage.

Once the deployment is complete, log into the OpenShift Web Console and complete an addition configuration step. Go to the logging project, click on the Kubana route, and accept the SSL exception in your browser.

Creation of additional users

To create additional (non-admin) users in your environment, login to your master server(s) via SSH and run:
htpasswd /etc/origin/master/htpasswd mynewuser

Additional OpenShift Configuration Options

You can configure additional settings per the official (OpenShift Enterprise Documentation).