Skip to content

Commit

Permalink
[hci][esxi] changes to use esxi as hci node
Browse files Browse the repository at this point in the history
  • Loading branch information
m4r1k committed Mar 28, 2020
1 parent f338beb commit d7149a1
Show file tree
Hide file tree
Showing 9 changed files with 111 additions and 17 deletions.
2 changes: 2 additions & 0 deletions README.md
Expand Up @@ -24,6 +24,8 @@ Usually changes between clouds can be found in the following files
## Folder structure
```
├── deployment_wrappers => Wrapper directory including NFVi deployment and initialization scripts (e.g. Docker images, import baremetal environment, etc)
├── hci => Folder with script to run HCI node on RHEL
├── hci-esxi => Folder with Readme, VMX and script to run HCI node on VMware ESXi
├── overcloud => Overcloud main directory for custom templates and environment configuration
│   ├── environments => Templates and configurations for the platform (network, SSL, storage, tuning, and extraconfig hooks mapping)
│   ├── extraconfig => Pre and post-puppet customization hooks templates and scripts
Expand Down
2 changes: 2 additions & 0 deletions hci-esxi/README.md
@@ -0,0 +1,2 @@
[Emulating an SSD Virtual Disk in a VMware Environment](https://www.virtuallyghetto.com/2013/07/emulating-ssd-virtual-disk-in-vmware.html)
[Sample configuration of virtual switch VLAN tagging](https://kb.vmware.com/s/article/1004074)
11 changes: 11 additions & 0 deletions hci-esxi/README_performance.md
@@ -0,0 +1,11 @@
# Userful Knowledge Base
- [Performance Best Practices for VMware vSphere 6.7](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-67-performance-best-practices.pdf)
- [Best Practices for Performance Tuning of Telco and NFV Workloads in vSphere](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-tuning-telco-nfv-workloads-vsphere-white-paper.pdf)
- [VMware vCloud NFV 3.0](https://docs.vmware.com/en/VMware-vCloud-NFV/3.0/vmware-vcloud-nfv-30.pdf)
- [Hyperthreading and ESXi Hosts](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-3362A2E9-AB03-4C10-B9A1-3E6CA78D399B.html)
- [What is PreferHT and When To Use It](https://blogs.vmware.com/vsphere/2014/03/perferht-use-2.html)
- [Assign a Virtual Machine to a Specific Processor](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-F40F901D-C1A7-43E2-90AF-E6F98C960E4B.html)
- [Backing Guest vRAM with 1GB Pages](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-F0E284A5-A6DD-477E-B80B-8EFDF814EE01.html)
- [Associate Virtual Machines with Specified NUMA Nodes](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-A80A6337-7B99-48C8-B024-EE47E2366C1B.html)
- [Checking cpuinfo information on an ESXi host](https://kb.vmware.com/s/article/1031785)
- [VMware vSphere - Why checking NUMA Configuration is so important!](https://itnext.io/vmware-vsphere-why-checking-numa-configuration-is-so-important-9764c16a7e73)
33 changes: 33 additions & 0 deletions hci-esxi/proxy-vbmc-esxi.sh
@@ -0,0 +1,33 @@
#!/bin/bash

# Make sure Subscription is done
# Make sure packages can be downloaded

subscription-manager attach --pool 8a85f99c6e417e37016e6c2fb8180766

subscription-manager repos \
--disable "*" \
--enable rhel-8-for-x86_64-baseos-rpms \
--enable rhel-8-for-x86_64-appstream-rpms \
--enable rhel-8-for-x86_64-supplementary-rpms \
--enable codeready-builder-for-rhel-8-x86_64-rpms

dnf makecache

dnf upgrade -y

dnf install -y ipmitool lsof tcpdump vim git

dnf install -y python3-virtualenv python3-libvirt libvirt-devel gcc make
virtualenv /root/vBMC
source /root/vBMC/bin/activate
pip install --upgrade pip
pip install virtualbmc==1.6.0
firewall-cmd --zone=public --permanent --add-port=623/udp
firewall-cmd --reload

# Do not accept forwarded locale
sed -e 's/^\(AcceptEnv LANG.*\)/#\1/g' -e 's/^\(AcceptEnv LC_.*\)/#\1/g' -i /etc/ssh/sshd_config
systemctl restart sshd

echo "## REBOOT THE vBMC PROXY PLEASE ##"
35 changes: 35 additions & 0 deletions hci-esxi/vBMC.sh
@@ -0,0 +1,35 @@
#!/bin/bash

_ESXiCONNECTION="--libvirt-uri esx://192.168.178.13/?no_verify=1 --libvirt-sasl-username root --libvirt-sasl-password Diablo1108!"

source /root/vBMC/bin/activate

ip addr show ens192 | grep -q "192.168.178.25/24" || ip addr add 192.168.178.25/24 dev ens192
ip addr show ens192 | grep -q "192.168.178.26/24" || ip addr add 192.168.178.26/24 dev ens192
ip addr show ens192 | grep -q "192.168.178.27/24" || ip addr add 192.168.178.27/24 dev ens192
ip addr show ens192 | grep -q "192.168.178.28/24" || ip addr add 192.168.178.28/24 dev ens192
ip addr show ens192 | grep -q "192.168.178.29/24" || ip addr add 192.168.178.29/24 dev ens192
ip addr show ens192 | grep -q "192.168.178.30/24" || ip addr add 192.168.178.30/24 dev ens192

vbmcd

vbmc show CEPH0 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.25 --port 623 ${_ESXiCONNECTION} CEPH0
vbmc show CEPH1 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.26 --port 623 ${_ESXiCONNECTION} CEPH1
vbmc show CEPH2 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.27 --port 623 ${_ESXiCONNECTION} CEPH2
vbmc show CTRL0 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.28 --port 623 ${_ESXiCONNECTION} CTRL0
vbmc show CTRL1 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.29 --port 623 ${_ESXiCONNECTION} CTRL1
vbmc show CTRL2 2>/dev/null || vbmc add --username root --password calvin --address 192.168.178.30 --port 623 ${_ESXiCONNECTION} CTRL2

vbmc show CEPH0 | grep status | grep -q running || vbmc start CEPH0
vbmc show CEPH1 | grep status | grep -q running || vbmc start CEPH1
vbmc show CEPH2 | grep status | grep -q running || vbmc start CEPH2
vbmc show CTRL0 | grep status | grep -q running || vbmc start CTRL0
vbmc show CTRL1 | grep status | grep -q running || vbmc start CTRL1
vbmc show CTRL2 | grep status | grep -q running || vbmc start CTRL2

ipmitool -H 192.168.178.25 -U root -P calvin -p 623 -I lanplus power status
ipmitool -H 192.168.178.26 -U root -P calvin -p 623 -I lanplus power status
ipmitool -H 192.168.178.27 -U root -P calvin -p 623 -I lanplus power status
ipmitool -H 192.168.178.28 -U root -P calvin -p 623 -I lanplus power status
ipmitool -H 192.168.178.29 -U root -P calvin -p 623 -I lanplus power status
ipmitool -H 192.168.178.30 -U root -P calvin -p 623 -I lanplus power status
16 changes: 13 additions & 3 deletions overcloud/environments/30-storage-environment.yaml
Expand Up @@ -25,19 +25,29 @@ parameter_defaults:
# Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
GnocchiBackend: rbd

# Here the Ceph Storage node config
# Here the Ceph Storage node config under Linux HCI
# NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
# sda 8:0 0 50G 0 disk
# ├─sda1 8:1 0 1M 0 part
# └─sda2 8:2 0 50G 0 part /
# sdb 8:16 0 8G 0 disk <- Optane NVME
# sdc 8:32 0 75G 0 disk <- SSD Data
# sdd 8:48 0 75G 0 disk <- SSD Data

# Here the Ceph Storage node config under ESXi HCI
# NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
# sda 8:0 0 50G 0 disk
# ├─sda1 8:1 0 1M 0 part
# └─sda2 8:2 0 50G 0 part /
# sdb 8:16 0 75G 0 disk <- SSD Data
# sdc 8:32 0 75G 0 disk <- SSD Data
# nvme0n1 259:0 0 8G 0 disk <- Optane NVME

CephAnsibleDisksConfig:
devices:
- /dev/sdc
- /dev/sdd
- /dev/sdb
- /dev/sdc
- /dev/nvme0n1
osd_scenario: lvm
# It is recommended that the block.db size isn’t smaller than 4% of block.
# For example, if the block size is 1TB, then block.db shouldn’t be less than 40GB.
Expand Down
16 changes: 8 additions & 8 deletions overcloud/environments/99-extraconfig.yaml
Expand Up @@ -10,14 +10,14 @@ resource_registry:
OS::TripleO::ComputeOvsDpdkSriovRTExtraConfigPre: ../extraconfig/pre-config/compute_realtime.yaml

# Compute Deterministic specific tuning
OS::TripleO::ComputeOvsDpdkExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeDualOvsDpdkExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeDualSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeOvsDpdkSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml

# Ceph SSD Workaround
OS::TripleO::CephStorageExtraConfigPre: ../extraconfig/pre-config/ceph_ssd_workaround.yaml
OS::TripleO::ComputeOvsDpdkExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeDualOvsDpdkExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeDualSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml
OS::TripleO::ComputeOvsDpdkSriovExtraConfigPre: ../extraconfig/pre-config/compute_deterministic.yaml

# Ceph SSD Workaround only required under Linux HCI
# OS::TripleO::CephStorageExtraConfigPre: ../extraconfig/pre-config/ceph_ssd_workaround.yaml

# Controller Role Post-Configuration hook for the Heat cache and to Beast RadosGW frontend
# Queens TripleO RGW template hardcodes civetweb effectively disabling Beast
Expand Down
10 changes: 5 additions & 5 deletions overcloud/overcloud_images.yaml
@@ -1,6 +1,6 @@
# Generated with the following on 2020-03-13T12:34:44.844162
# Generated with the following on 2020-03-20T10:34:48.318864
#
# openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination=10.0.10.10:8787 --prefix=openstack- --tag-from-label {version}-{release} --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --output-env-file=../overcloud/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml -r /home/stack/roles-data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/sshd-banner.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/cinder-backup.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml -e ../overcloud/environments/10-commons-parameters.yaml -e ../overcloud/environments/20-network-environment.yaml -e ../overcloud/environments/30-storage-environment.yaml -e ../overcloud/environments/40-fencing.yaml -e ../overcloud/environments/50-keystone-admin-endpoint.yaml -e ../overcloud/environments/60-openstack-neutron-custom-configs.yaml -e ../overcloud/environments/65-openstack-nova-custom-configs.yaml -e ../overcloud/environments/70-ovs-dpdk-sriov.yaml -e ../overcloud/environments/99-extraconfig.yaml -e ../overcloud/environments/99-server-blacklist.yaml
# openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination=10.0.10.10:8787 --prefix=openstack- --tag-from-label {version}-{release} --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --output-env-file=../overcloud/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml -r /home/stack/roles-data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/sshd-banner.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/cinder-backup.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml -e ../overcloud/environments/10-commons-parameters.yaml -e ../overcloud/environments/20-network-environment.yaml -e ../overcloud/environments/30-storage-environment.yaml -e ../overcloud/environments/40-fencing.yaml -e ../overcloud/environments/50-keystone-admin-endpoint.yaml -e ../overcloud/environments/60-openstack-neutron-custom-configs.yaml -e ../overcloud/environments/60-openstack-nova-custom-configs.yaml -e ../overcloud/environments/60-openstack-glance-custom-configs.yaml -e ../overcloud/environments/70-ovs-dpdk-sriov.yaml -e ../overcloud/environments/99-extraconfig.yaml -e ../overcloud/environments/99-server-blacklist.yaml
#

parameter_defaults:
Expand Down Expand Up @@ -57,12 +57,12 @@ parameter_defaults:
DockerNeutronMetadataImage: 10.0.10.10:8787/rhosp13/openstack-neutron-metadata-agent:13.0-115
DockerNeutronSriovImage: 10.0.10.10:8787/rhosp13/openstack-neutron-sriov-agent:13.0-112
DockerNovaApiImage: 10.0.10.10:8787/rhosp13/openstack-nova-api:13.0-114
DockerNovaComputeImage: 10.0.10.10:8787/rhosp13/openstack-nova-compute:13.0-129
DockerNovaComputeImage: 10.0.10.10:8787/rhosp13/openstack-nova-compute:13.0-129.1584463509
DockerNovaConductorImage: 10.0.10.10:8787/rhosp13/openstack-nova-conductor:13.0-113
DockerNovaConfigImage: 10.0.10.10:8787/rhosp13/openstack-nova-api:13.0-114
DockerNovaConsoleauthImage: 10.0.10.10:8787/rhosp13/openstack-nova-consoleauth:13.0-112
DockerNovaLibvirtConfigImage: 10.0.10.10:8787/rhosp13/openstack-nova-compute:13.0-129
DockerNovaLibvirtImage: 10.0.10.10:8787/rhosp13/openstack-nova-libvirt:13.0-134
DockerNovaLibvirtConfigImage: 10.0.10.10:8787/rhosp13/openstack-nova-compute:13.0-129.1584463509
DockerNovaLibvirtImage: 10.0.10.10:8787/rhosp13/openstack-nova-libvirt:13.0-134.1584463509
DockerNovaMetadataImage: 10.0.10.10:8787/rhosp13/openstack-nova-api:13.0-114
DockerNovaPlacementConfigImage: 10.0.10.10:8787/rhosp13/openstack-nova-placement-api:13.0-113
DockerNovaPlacementImage: 10.0.10.10:8787/rhosp13/openstack-nova-placement-api:13.0-113
Expand Down
3 changes: 2 additions & 1 deletion undercloud/undercloud.conf
Expand Up @@ -98,7 +98,8 @@ local_subnet = ctlplane-subnet

# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = eth1
# Linux HCI eth1 while ESXi ens224
local_interface = ens224

# MTU to use for the local_interface. (integer value)
local_mtu = 1500
Expand Down

0 comments on commit d7149a1

Please sign in to comment.