Skip to content

OSVDC Series: Compute Host Build with CentOS 7 1511

Jump to bottom
RHarmonson edited this page Jul 27, 2017 · 59 revisions

OSVDC: Compute Host Build with CentOS 7 1511

Revised: April 8, 2016

Article 7 of the Open Source Virtual Data Center Series

Prerequisites

Access to the internet for use by the Compute host. Also, the section titled Install & Update CentOS directs the use of my CentOS 7 1511 Minimal x86_64 Base Installation Guide. A workstation or laptop to use as a SSH client is very useful if not a requirement per se.

Assemble Hardware

If you have no or little experience in assembling computer components, please review the following for help. Youtube is an excellent resource, too. Prevention of ESD is very important!

Update System Board BIOS (firmware)

Per Supermicro's website, flashing or upgrading the system board voids the warranty! If you do not have the most current BIOS, download the BIOS update here:

http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLL_-F.cfm

  • Select "Update Your BIOS"

After download, open the archive and read and follow the instructions carefully.

I updated with x10sllp5.424 which results with the BIOS displaying version 3.0 build date 04/24/2015.

Update "BMC/IPMI Firmware"

Update the BMC/IPMI firmware if you do not have the current by downloading the archive from here:

http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLL_-F.cfm

  • Select "BMC/IPMI Firmware"

After download, open the archive and read and follow the instructions carefully.

I updated with SMT_X10_192.bin which results with the BIOS displaying for IPMI version 1.92.

While you are waiting for the firmware to be updated, consider reading Best Practices BMC Security.

Update Chelsio

Drivers, ROMs, and documentation can be found at Chelsio Support website.

I am using the DOS download version 1.0.0.85, dated 07/07/2015, and with the file name Chelsio-Uboot-1.0.0.85.zip which uses the firmware file t5fw-1.13.32.0.bin. Results with boot ver. 1.0.0.85, firmware ver. 1.13.32.0, and config ver. 1.0.0.85.

Since the flash is more convoluted than most, I am providing the instructions as found in the doc/README.txt file. Please review before flashing to verify nothing has changed.

I. Preparing USB flash drive
-----------------------------

This document assumes that you are using an USB flash drive as a storage 
media for the necessary files. Follow the steps below to prepare the drive:

a. Create a DOS bootable USB flash drive.
b. Create a directory "CHELSIO" on USB flash drive.
c. If you haven't done already, download Chelsio-Uboot-x.x.x.xx.zip from Chelsio
   Download Center, service.chelsio.com
d. Unzip the downloaded package and change your working directory to "OptionROM"
   directory. 
e. Copy all the files and place them in the CHELSIO directory created on the USB 
   flash drive.
f. Plug-in the USB flash drive in the system on which the Chelsio CNA is 
   installed.
g. Reboot the system and enter the system BIOS setup.
h. Make the USB flash drive as the primary boot device. 
i. Save the changes and exit.

II. Legacy 
----------

a. Configure the system having Chelsio CNA to boot in Legacy mode. 

b. Once the system boots from the USB flash drive, change your working directory
   to CHELSIO directory:

   C:\>cd CHELSIO

c. Run the following command to list all Chelsio CNAs present in the 
   system. The list displays a unique index for each CNA found.

   C:\CHELSIO>cfut4 -l

d. Delete any previous version of Option ROM flashed onto the CNA:

   C:\CHELSIO>cfut4 -d <idx> -xb 
   
Here, idx is the CNA index found in step (c) 

e. Delete any previous firmware using the following command:

   C:\CHELSIO>cfut4 -d <idx> -xh -xf 
  
f. Delete any previous Option ROM settings:

   C:\CHELSIO>cfut4 -d <idx> -xc

g. Run the following command to flash the appropriate firmware 
   (t5fw-x.xx.xx.x.bin for T5 adapters;t4fw-x.xx.xx.x.bin for T4 adapters). 

   C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin
   
h. Flash the Option ROM onto the Chelsio CNA using the following command:

   C:\CHELSIO>cfut4 -d <idx> -ub cubt4.bin  

Here, "cubt4.bin" is the unified option ROM image file present in the CHELSIO 
directory.   
 
i. Flash the default boot configuration file.  

   C:\CHELSIO>cfut4 -d <idx> -uc bootcfg

j. Reboot the system for changes to take effect.

Burn-in

It is a good idea to leave the assembled system running for a day or two to burn-in. You can, also, use software to stress test the system but I have not found a suitable solution that does not require installing an operating system first. At a later date I may explore Phoronix Test Suite. It looks to have potential.

Obtain CentOS

I am using Centos 7 build 1511 which is equivalent to RHEL 7.2. The most current release and checksums can be found here:

https://www.centos.org/download/

Release 1511 at this time is here:

http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso

Documentation and help are found on CentOS wiki and forums:

Wiki: https://wiki.centos.org/

Forum: https://www.centos.org/forums/

Install & Update CentOS

Please use my guide found at the URL below for the initial installation and configuration of CentOS 7. Note the setup of the EPEL repository in the guide is not a prerequisite and can be disregarded.

https://github.com/rharmonson/richtech/wiki/CentOS-7-1511-Minimal-x86_64-Base-Installation-Guide

You can install CentOS using a CD, USB flash drive, or Supermicro's BMC/IPMI. I will be using IPMI.

BMC/IPMI

Using the Intelligent Platform Management Interface (IPMI) is not the easiest nor fastest method to install an operating system. However, once it is setup and working, IPMI, provides out-of-band management via redirected consoles (KVM) and devices even when the operating system faults or services like SSH or RDP are unavailable.

Configure IPMI using BIOS:

  1. Connect an ethernet cable to the dedicated network interface
  2. Power-on or reboot the system
  3. Select the DEL key at the Supermicro splash screen to enter the system BIOS
  4. Select the "IPMI" menu
  5. Select "BMC Network Configuration"
  6. Select "Update IPMI LAN Configuration" to toggle No to Yes
  7. Enter the IP address 192.168.101.2
  8. Enter the subnet mask 255.255.255.0
  9. Enter the default gateway 192.168.101.254
  10. Select F4 to save and reboot

IPMI User Manual

ftp://ftp.supermicro.com/utility/IPMIView/IPMIView20.pdf

Using IPMI's Remote Control requires the use of a share to mount an installation ISO.

To complete an operating system installation using IPMI:

  1. Login to IPMI; https://ip_address, e.g. 192.168.101.2, using user ADMIN, password ADMIN. Case sensitive!
  2. Mount installation image; select "Virtual Media," "CD-ROM Image" and specify host, path to image, credentials, then click "Mount" (See Example below)
  3. Open remote control; select "Remote Control," "Console Redirection", then "Launch Console" (Java)
  4. Restart server; select "Remote Control," "Power Control," then "Power Cycle Server"
  5. At the Supermicro splash screen, select F11 to display boot device menu"
  6. Select virtual CDROM (ATEN Virtual CDROM YS0J)
  7. Install operating system
  8. Umount installation image; select "Virtual Media," "CD-ROM Image," then click "Unmount"
  9. Complete!

Virtual Media Example

Assumption: You are using a windows workstation. If so, you are not permitted to browse to an ISO on the local file system but must use a CIFS or SMB share. Odd. Using windows, you can simply select a folder, right mouse click, and select "Share with." In the example below, I am using a CIFS share that permits guest read access. Otherwise, you will need to provide user credentials with the appropriate access.

  • Share host: 192.168.22.10
  • Path to image: \cifsshare\instmedia\isos\centos\7\1511\centos7min-1511.iso
  • User: guest
  • Password: (blank)

Security Consideration

Access to BMC/IPMI like all privileged services should be carefully restricted. Firewall dependencies and best practices per Supermicro for BMC/IPMI are found here:

http://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf

CentOS Configuration

After install and update of the base operating system, we have a number of tasks to complete prior to installation of oVirt.

Host Name

The first Compute host is named node1. Update and view the changes using the hostnamectl command.

[root@localhost ~]# hostnamectl set-hostname node1.mydomain.net
[root@localhost ~]# hostnamectl
   Static hostname: node1.mydomain.net
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: b2dc04fb1430417cad1042a40ac5189f
           Boot ID: a56313ff13e442b1b0e06edbd293277c
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.13.1.el7.x86_64
      Architecture: x86-64

Networking

Network interfaces are configured using nmtui, nmcli, or editing ifcfg files by hand. The use of nmtui and nmcli requires NetworkManager be enabled and started but nmtui is very intuitive and easy to use. However, I am not a fan of NetworkManager on minimal installations and question is usefulness in general. As such, I will be using vi to edit or hand-craft ifcfg files found /etc/sysconfig/network-scripts/.

If you followed my CentOS 7 1511 Minimal x86_64 Base Installation Guide, you may have already completed most of the work.

Management Interface

The management interfaces are enp6s0f0 (and, later, enp6s0f1). Note the settings BOOTPROTO=none and DEFROUTE=yes. Assuming you don't have or plan to have an external DNS host to resolve, specify DNS1=127.0.0.1 for use with dnsmasq. By "external" it is meant a DNS host that will not be a virtual machine hosted by this cluster.

[root@node1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-enp6s0f0
DEVICE=enp6s0f0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.101.11
PREFIX=24
GATEWAY=192.168.101.254
DEFROUTE=yes
DNS1=127.0.0.1
DNS2=8.8.8.8
DOMAIN=mydomain.net
PEERDNS=yes

Storage Interface

The storage interface is enp1s0f4. Note the use of PREFIX=30 and DEFROUTE=no.

[root@node1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-enp1s0f4
DEVICE=enp1s0f4
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.111.2
PREFIX=30
DEFROUTE=no

Name Resolution

Typically, if you have a DNS host that resides outside of the cluster, you need to add records for the Storage host, Compute hosts, and oVirt Management Engine. If not or if you intent is to run the DNS hosts within the cluster, a solution using a combination of /etc/hosts and `dnsmasq' due to a "Chicken and the Egg" scenario at initial boot of the cluster. I am, also, using the hosts file versus DNS to remedy not having a 10 Gb switch. Note the entry for the second node and engine in preparation for later.

root@node1 ~]# vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.111.1 nasv.mydomain.net nasv
192.168.101.11 node1.mydomain.net node1
192.168.101.12 node2.mydomain.net node2
192.168.101.21 eng.mydomain.net eng

The dnsmasq package should already be installed, but it needs to enabled and started.

[root@node1 ~]# systemctl enable dnsmasq
Created symlink from /etc/systemd/system/multi-user.target.wants/dnsmasq.service to /usr/lib/systemd/system/dnsmasq.service.
[root@node1 ~]# systemctl start dnsmasq

During our task above configuring DNS with 127.0.0.1 and 8.8.8.8 on the management interface, /etc/resolv.conf should have resulted with:

# Generated by NetworkManager
search mydomain.net
nameserver 127.0.0.1
nameserver 8.8.8.8

It is advisable but not required to add an additional entry for a third DNS host. Using vi, add nameserver 8.8.4.4.

[root@node1 network-scripts]# cat /etc/resolv.conf
# Generated by NetworkManager
search intranet.harmonson.net
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4

Warning!

If name resolution is not correctly working, NFSv4 should not function as desired. For example when testing a NFS export and creating a file, you may observer owner nobody:nobody instead of vdsm:kvm. If using NFSv3, there is no such requirement.

Compute Host #2

At this point you may proceed to build either the second Compute host prior to oVirt installation or return later after completing the next article in the series. The second Compute host installation is a repeat and rinse of the above except that the interface configurations and /etc/hosts file will use these values given below.

  • enp6s0f0: 192.168.101.12/24
  • enps0f4: 191.168.112.2/30
  • nasv: 192.168.112.1/30 <-- /etc/hosts

##Multipathing Errors

After the installation of oVirt, spurious kernel: device-mapper: table: 253:3: multipath: error getting device errors are generated. The fix is to blacklist the device using its wwid in /etc/multipath.conf. Determine the wwid of the device by executing multipath -v2. Mine is given as an example below, but each device will have a unique wwid. To effect the change, you will need to either reboot or execute multipath -F.

[root@node1 yum.repos.d]# cat /etc/multipath.conf
# VDSM REVISION 1.3

defaults {
    polling_interval            5
    no_path_retry               fail
    user_friendly_names         no
    flush_on_last_del           yes
    fast_io_fail_tmo            5
    dev_loss_tmo                30
    max_fds                     4096
}

# Remove devices entries when overrides section is available.
devices {
    device {
        # These settings overrides built-in devices settings. It does not apply
        # to devices without built-in settings (these use the settings in the
        # "defaults" section), or to devices defined in the "devices" section.
        # Note: This is not available yet on Fedora 21. For more info see
        # https://bugzilla.redhat.com/1253799
        all_devs                yes
        no_path_retry           fail
    }
}

# Enable when this section is available on all supported platforms.
# Options defined here override device specific options embedded into
# multipathd.
#
# overrides {
#      no_path_retry           fail
# }

blacklist {
        wwid SanDisk_Ultra_Fit_4C531001600429120084-0:0
}

Reference: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/config_file_blacklist.html

Additional References

RHEL 7 General Storage Administration Guide

RHEL 7 Networking Guide

Next

Next article in the series is Virtualization Services with oVirt 3.6 and oVirt Engine Appliance.

Clone this wiki locally