Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Configure OSA Multi node Environment
Please refer to the OSA Install guide for detailed instructions to setup a multi-node OpenStack environment
The minimum deployment recommended is as follows:
- Three infrastructure (a.k.a. controller) nodes
- One logging node
- One compute node
One of the infrastructure hosts (infra1) doubled as the deployment host. The deployment host is where the openstack-ansible code is cloned, ansible is installed, and the playbooks are run to accomplish the deployment.
All five nodes used in the deployment are Ubuntu 14.04 Server VMs running under KVM on a bare-metal host that also was running Ubuntu 14.04 Server. OSA deployments in production would generally occur on bare metal hosts rather than VMs but this document refers to a complete VM based deployment. KVM was configured for the following virtual neworks:
- "data" network, NAT-enabled, DHCP-enabled, addresses 192.168.1.200-254 set aside for static IPs
- "system" network, no NAT, no DHCP
Each VM was configured to have the following resources:
- 100 GB Disk
- 4096 MB RAM
- 2 CPU
- 2 vNICs
- eth0 was on the "data" network
- eth1 was on the "system" network
Node Network Configuration
The openstack-ansible installation document describes a general network configuration for each node. This configuration must be done manually on each node prior to playbooks being run. It is important to note, however, that the description in the installation guide assumes that the installer will be following best-practices using bonded interfaces. The deployment process detailed in this document assumes non-bonded interfaces.
A simplified diagram of the setup is shown, below:
Here is a sample /etc/network/interfaces file, taken from infra1:
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The data network interface auto eth0 iface eth0 inet static address 192.168.1.201 gateway 192.168.1.1 netmask 255.255.255.0 dns-nameservers 192.168.1.1 22.214.171.124 126.96.36.199 # The system network interface auto eth1 iface eth1 inet manual # Container management VLAN interface iface eth0.10 inet manual vlan-raw-device eth0 # OpenStack Networking VXLAN (tunnel/overlay) VLAN interface iface eth1.30 inet manual vlan-raw-device eth1 # Storage network VLAN interface (optional) iface eth0.20 inet manual vlan-raw-device eth0 # Container management bridge auto br-mgmt iface br-mgmt inet static bridge_stp off bridge_waitport 0 bridge_fd 0 offload-sg off # Bridge port references tagged interface bridge_ports eth0.10 address 172.29.236.11 netmask 255.255.252.0 dns-nameservers 188.8.131.52 184.108.40.206 # OpenStack Networking VXLAN (tunnel/overlay) bridge auto br-vxlan iface br-vxlan inet static bridge_stp off bridge_waitport 0 bridge_fd 0 offload-sg off # Bridge port references tagged interface bridge_ports eth1.30 address 172.29.240.11 netmask 255.255.252.0 # OpenStack Networking VLAN bridge auto br-vlan iface br-vlan inet manual bridge_stp off bridge_waitport 0 bridge_fd 0 offload-sg off # Bridge port references untagged interface bridge_ports eth1 # Storage bridge (optional) auto br-storage iface br-storage inet static bridge_stp off bridge_waitport 0 bridge_fd 0 offload-sg off # Bridge port reference tagged interface bridge_ports eth0.20 address 172.29.244.11 netmask 255.255.252.0
By default, the file /etc/openstack_deploy/user_secrets.yml has no passwords. If this file isn't updated, OSA deployment will eventually fail due to a lack of permissions. This can be avoided by following the instructions: http://docs.openstack.org/developer/openstack-ansible/liberty/install-guide/configure-creds.html
Load Balancer, haproxy, and repo servers
Each node will use its Container Management bridge network during the deployment for various operations. For example, when a container on a node wants to do a pip install, it tries to contact the pypi server using the IP address that you define for internal_lb_vip_address in the /etc/openstack_deploy/openstack_user_config.yml file. In absence of a real load balancer IP, use the container management IP address of the haproxy server. In this document, infra1 serves as the haproxy server.
Now, for the haproxy server to work as a load balancer in this pip case, designate the infrastructure nodes as repo nodes. This will allow haproxy to be configured to forward pip requests to one of the infrastructure hosts as server.
To accomplish these things, ensure that the following parameters are configured in the /etc/openstack_deploy/openstack_user_config.yml file. Note that this is not the complete file. Snippets are shown for reference:
# # Define three shared infrastructure hosts: # shared-infra_hosts: infra1: ip: 172.29.236.11 infra2: ip: 172.29.236.12 infra3: ip: 172.29.236.13 # # Define haproxy host: # haproxy_hosts: infra1: ip: 172.29.236.11 # # Define three package repository hosts: # repo-infra_hosts: infra1: ip: 172.29.236.11 infra2: ip: 172.29.236.12 infra3: ip: 172.29.236.13 # # Define global overrides # global_overrides: internal_lb_vip_address: 172.29.236.11 external_lb_vip_address: 172.29.236.11
To enable logging during an ansible playbook run, edit /opt/openstack-ansible/playbooks/ansible.cfg and add a single line. For example:
log_path = /opt/openstack-ansible/playbooks/ansible.log
The openstack-ansible command accommodates different levels of verbosity. If you append '-v', '-vv', '-vvv', or '-vvvv' to the command to run a playbook you get increasing levels of verbosity in the output and in the logs. '-vvvv' is the highest level and includes connectivity information.
When a failure running a playbook occurs, ansible deposits a file in your ~ folder. This file can be used on the next playbook run to limit what ansible will redo. If used, openstack-ansible won't redo some operations that had completed successfully. The syntax for using this requires that you add something like the following to your command line:
The example, above, is following a failure while running the setup-hosts.yml playbook.
Note that some playbooks are nothing more than execution of a list of other playbooks. setup-hosts, for example, executes:
This example is illustrative of a couple of things that can be useful during debugging.
First, to reduce the time required to re-try things, you can simply re-execute the sub-playbook where the failure occurred instead of running all 3.
Second, some playbooks have compliments. If you look on disk, you can find both lxc-containers-create.yml and lxc-containers-destroy.yml. You can "start over" by running destroy and re-running create.
--limit is a command line option that can be used to reduce the scope of the execution of a playbook. Using the "destroy" example, above, one can limit the scope of lxc-containers-destroy.yml with the following example:
openstack-ansible lxc-containers-destroy.yml --limit keystone_all
This example would destroy all the keystone containers, leaving all other containers intact.