Automatically Building Scopes from Hypervisor Discovery

Building out scopes for discovery in TADDM can be a moving target. New systems are added and removed from the environment constantly. In order to address this, I have written a set of scripts that discovers hypervisors (System P HMC and VMware vCenter) and then automatically builds scope sets of child VM systems from those results by geography.

Prerequisites

The script that runs the discovery and controls the processing relies on a scope building script that can be found on the wiki page "Building Scope Sets from Discovered Systems". Download and install buildscope.jy before using this script.

There is a script that runs some basic error checking against the discovery logs for a particular run. You will need to download and install checklogs.sh and place it under $COLLATION_HOME/custom. The script works by issuing grep commands for specific error conditions.

Here are the error conditions along with the corresponding log file and grep query.

Error	Log	Grep
Ping failed	PingSensor-*	Ping failed for IP address .* on all ports
Port sensor found nothing	PortSensor-*	INFO sensor\.PortScanSensor .* found nothing
SSH session error	SessionSensor-*	CTJTP1160E The application cannot establish the following SSH session
.NET version failure	SessionSensor-*	CTJTP1135E The following text is the exit status: -2146232576
WMI error	SessionSensor-*	CTJTP1163E The following WMI session and SSH sessions cannot be established
svmon failure	AixComputerSystemSensor-*	CTJTD0809W Sensor cannot discover virtual memory configuration
lsof failure	GenericServerSensor-*	CTJTD0312E No Listening Processes Found
lspartition failure	HmcSensor-*	WARN processors.LspartitionProcessor - CTJTD0829E Error executing command:lspartition -dlpar

Here is sample output from the checklogs.sh script.

Checking SystemP HMC discovery log files for errors
Ping failed for 192.168.10.25
Ping failed for 192.168.10.26
Port sensor found nothing 192.168.234.65
SSH session error for 192.168.212.161
SSH session error for 192.168.212.162
SSH session error for 192.168.21.63
SSH session error for 192.168.4.31

If you would like to add an error condition, follow the pattern of the existing error conditions.

Installation

Download autodiscovery.sh and place it under $COLLATION_HOME/custom on a discovery or domain server. Edit the file and change the USER, PASSWORD, and PROFILE variables if appropriate.

Create the directory $COLLATION_HOME/custom/scopes. This is where the scope files will be held.

Configuring Hypervisor Scopes

The input to the entire process is the hypervisor scopes, which must be maintained manually. Currently, System P and vCenter are supported. Within a hypervisor type, you can add scopes for various geographies and the child system scopes will be created according to geo. For this, a naming convention is used for the scope files. The file names (and scope names) end with "_<geo>" where <geo> is the particular geography name. For example, a scope listing the vCenters for North America could be named scopes/vcenter_NA.scope.

For vCenter scopes, the file name must conform to the naming convention vcenter_<geo>.scope for the script to process it. The target IP addresses for the VMware vCenter should be put in these files.

For System P HMC scopes, the file name must conform to the naming convention systemp_hmc_<geo>.scope for the script to process it. The target IP addresses for the HMC should be put in these files.

Create the scope files for the proper hypervisors under the scopes directory. For details about the format of a scope file, see the TADDM user's guide. For the first line in each scope file, add a comment with a unique scope name to use. Here is an example scope file named vcenter_NA.scope.

# vCenter NA
11.16.313.262,,kristina.ibm.com
11.16.313.263,,britney.ibm.com
11.16.313.264,,lindsay.ibm.com

When the files are loaded, the scope name in the comments on the first line will be used, but any blanks will be replaced by under-scores.

How it Works

The general flow of the script is as follows.

1. System P HMC scopes are loaded into TADDM from scopes/systemp_hmc_<geo>.scope files.
2. Discovery is run against all HMC targets using the discovery profile specified in the script. The script goes into a sleep cycle waiting for the discovery to complete.
3. The buildscopes.jy script is executed to build SystemP LPAR scopes by geography. Those scopes are then automatically loaded into TADDM. Discovery is not started against those targets.
4. VMware vCenter scopes are loaded into TADDM from scopes/vcenter_<geo>.scope files.
5. Discovery is run against all vCenter targets using the discovery profile specified in the script. The script goes into a sleep cycle waiting for the discovery to complete.
6. The buildscopes.jy script is executed to build Windows VM scopes by geography. Those scopes are then automatically loaded into TADDM. Discovery is not started against those targets.
7. The buildscopes.jy script is executed to build Linux VM scopes by geography. Those scopes are then automatically loaded into TADDM. Discovery is not started against those targets.

Scope names

For VMware, the scope names loaded into TADDM will be named "VMware Windows guests $geo" and "VMware Linux guests $geo". The scope file will be named vmware_guests_win_$geo.scope and vmware_guests_linux_$geo.scope and found under custom/scopes.

For System P, the scope names loaded into TADDM will be named "SystemP LPARs $geo". The scope file will be named systemp_lpar_$geo.scope and found under custom/scopes.