Fetching latest commit…
Cannot retrieve the latest commit at this time.


install azure cli client

Configure setting

The following outlines the settings that need to be configured. Please ensure that your groupname is unique as this is used as public dns name for the console

resource_group_name: ivancloud

## Azure AD user. ad_username: XXXXXXXXXXXXXXXXXXX

### Azure AD password


### this is your azure subscription id


## user to login to the jump host. this user will only be created on the jumphost

adminUsername: ivan

## user pwd for jump host

## Password for the jump host

adminPassword: XXx_please_change_me_xXX

##### Public key for jump host

sshkey: ssh-rsa XXXXXXXXXXXXX

#### subscription information

rh_subcription_user: XXXXXXXXXXX

rh_subcription_pass: XXXXXXXXXXX

openshift_pool_id: XXXXXXXXXXX

For the azure creditials, i recommend created a "ansible" user instead of using your existing adming user. The following steps can be followed to perform this

  • After creating a user as outlined in the above instructions. in the azure portal. go "settings"->"administrators", and then add user

Update ansible,

This ansible script uses features only found in the latest version on ansible (notable the azure plugin and the gatewayed feature ) upgrade to ansible

Note: After upgrade to ansible 2.2 i encountered issues where the azure module attempts to pass two params to a function that expects one. this was resolved by upgrade the phyton msresrt module. " sudo pip install msrest --upgrade"

this upgrade from msrest 0.4.0 to 0.4.1 - pip install ansible - pip install msrestazure --upgrade - pip install msrest --upgrade

See for info how to configure the module

run script

Update the group_vars/all variable. The following params exist. The script will support more masters in the future. For now the script installs 1 master, 1 infra, x number of nodes. the nodes and infra are get labels which corrospond the tags in azure (consistent)

  • ansible-playbook --forks=50 -i inventory playbooks/setupeverything.yml

Script modules

The scripting has now been split into more granular operations. The playbook setupeverything inlcudes has 3 main playbooks


Simply Provisions the basic infrastructure base on the Node Arrays defined in the group_vars/all file. This includes loadbalancer , availabilitySets and multiple subnets

### perform_prereqs.yml Configure all outline prereqs from the documention. Addition configures storage based on best practises to ensure if storage of logs, etc, openshift.volumes is exceeded it does not corrupt the rest of the node. Configures docker and installs required base packages


Creates dynamic ansible hosts file based on the contents of groups_var/all and the contents of the config under the prepare_multi roles

scaling out

After the initial installation has been run and openshift is up and runnin it is possible to had extra node to the cluster. Add the additional node definition to the groups_var/all file and run the ansible script

  • ansible-playbook --forks=50 playbooks/provision_infrastructure.yml This script is whats run in the playbook playbooks/setupeverything.yml

configuration of nodes

Under the groups/all there is a list of vms that will get created. This list also has a attribute called tag which. The values get set to azure tags and also openshift node labels

  • jump node : this is required. all actions are performed via this node. This node needs to come up first before any of the other nodes can be created. this is because we generate a key on the jump node which is distributed to all subsequent nodes. For convience sake the LB infront of the master is also placed on this node.(opens port 8443 and 22). In the future this will be a azure loadbalancer
  • master: we create 3 of these. no ports facing the outside. Loadbalanced by the jump node
  • infra node: currently hosts the registry and the router. not tested with multiple infra nodes yet. should work
  • node: bunch of nodes to host applications

Post install

The installation of openshift is performed on the jump node. The local ansible script (setup_multimaster) execute another ansible script on the jump host(advanced install). The host file has already been placed under /etc/ansible/host by this stage and is dynamically build based on the template sitting under

  • playbooks/roles/prepare_multi/templates/hosts.j2 The execution of the advanced install is executed in async manner (to avoid timeouts as the advanced install can take time ). The local ansible script(setup_multimaster) polls the jump host for completion.

Once installation is complete the following is configured

  • router ( is not deployed by default because the region=infra is not used)
  • Register: (is not deployed by default because the region=infra is not used)
  • metrics,
  • logging : waits untils everything is ready before scaling out the fluentd nodes ### example node setup under groupvars/all ## Example group_vars/all file

The following snippet is a brief example of how the ansible scripts are configured to bring up a set of nodes

    name: jumphost1
      region: "{{ region }}"
      zone: jumphost
      stage: jumphost
      type: jumphost

    name: master1
      region: "{{ region }}"
      zone: zone1
      type: infra

    name: infranode1
      region: "{{ region }}"
      zone: zone1
      infra: "true"
      type: core
      infratype: registry
      mbaas_id: mbaas1
    name: infranode2
      region: "{{ region }}"
      zone: zone2
      infra: "true"
      type: core
      mbaas_id: mbaas2
    name: infranode3
      region: "{{ region }}"
      zone: zone3
      infra: "true"
      type: core
      mbaas_id: mbaas3
    name: node1
      region: "{{ region }}"
      zone: zone1
      infra: "false"
      stage: dev
      type: apps
    name: node2
      region: "{{ region }}"
      zone: zone2
      infra: "false"
      stage: dev
      type: apps

Required params

To setup the following params are required to be added to the group_vars/all file

  • resource_group_name: ose86
  • subscriptionID: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  • adminUsername: userforjumphost
  • adminPassword: passwordforjumphost
  • sshkey : (public sshkey)
  • rh_subcription_user
  • rh_subcription_pass
  • openshift_pool_id

Example All Files

Under the directory all_example is a collection of sample all files. They are supplied as guidance on how to setup different variants of a openshift cluster

  • all_mega_cluster : A example of a production like cluster. wwith 3 masters, 3 infra nodes, 2 dev nodes and 2 production nodes
  • all_rhmap_core: A example demostrating a single master cluster that is labeled wit the expectation of RHMAP core been deployed ontoop (there is supplied playbooks to deploy RHMAP core, note you need relevent subscriptions )
  • all_multi_master: a example to demostration how to bring up a multi master cluster
  • all_single_master_small: A example of a bare-min cluster. with 1 master, 1 infra node, 1 application node

Debugging nodes

to debug stuff on nodes your first need to ssh to the jump node. ssh $adminUsername@jumpnodeip. The jump node ip should be written to groups/all file(only for later debugging purposes) but it is also writen to console as the ansible script is been executed. the jump nodes acts as a gateway for all vms created in azure. once logged into the jump node the user $adminUsername should have preconfigured ssh acces to all openshift nodes. Simply execute ssh master1 . The user $adminUsername also has sudo rights on all machines

monitoring openshift install

Since the openshift install is execute async on the jump node we cannot see the progress of the install. To following the progress of the install, You can login using your user ivan@ and tail the log file /tmp/ansible.log. TODO: write ansible check to distinguish between failure and success of the install.