LBNL NHC version 1.5

What's Changed in 1.5

New Features

• New command-line options:
  • -V - Display version information; may be specified multiple times for increasing detail
  • -j - Use JSON object format for syslog() payload
  • -mhost - Masquerade as host; i.e., force use of host as the hostname rather than the canonical hostname.
  • -p - Permanently offline nodes; nodes taken offline will be ignored by future nhc runs and will require admin intervention to return to service
  • -rresmgr - Specify resmgr as the resource manager (e.g., slurm) or "none" for standalone operation.
• Several new checks are available out-of-the-box, including ones for NVIDIA GPUs via SMI and DCGM, Kubernetes, and more!
• nhc-wrapper can now wrap co-located executables that need not be in the default/current $PATH.
• Default permissions have been adjusted in the RPM specfile to allow normal users to access the scripts directory and the check modules within. This is build-time configurable (via rpmbuild --without useraccess ...).
• The subprocess management and watchdog timer subsystems have been completely revamped to provide reliable and consistent oversight of child processes and to fix a timer conflict bug in previous versions.
• By popular demand, concurrency handling via a runtime process ID (PID) file has been implemented. PID files are context-specific and stored in ${RUNDIR}/nhc/NHC.context.pid by default. (The value for $RUNDIR is automatically detected.)
• When running in "verbose" mode (-v, VERBOSE=1), check numbers and totals are displayed as each check is run.
• check_file_stat() now supports the "birth timestamp" if available (via -B/-b) as well as the SELinux context (via -c, only available on Linux and when enabled).
• NHC's configuration file, specified via -c or the $CONFFILE setting, no longer needs to point to a regular file; valid values now include named pipes/FIFOs, AF_UNIX sockets, PTYs/TTYs, process substitutions, etc. (Example: nhc -avl- -c <( egrep -hv '^[[:space:]]*(#|$)' /etc/nhc/nhc*.conf ))
• Support was added for taking certain directory/path settings from namespaced environment variables:
  • High-Level Settings
    • NHC_CFG_GLOBAL - The complete path- and filename to a global settings file, used instead of /etc/sysconfig/nhc
    • NHC_CFG_PREFIX - Assume autoconf-style layout with --prefix=<value>
    • NHC_HOST - Override default $HOSTNAME value (/etc/hostname or /proc/sys/kernel/hostname)
  • Individual Directories
    • NHC_CFG_CONFDIR - Override default CONFDIR value (/etc/nhc)
    • NHC_CFG_INCDIR - Override default INCDIR value ($CONFDIR/scripts)
    • NHC_CFG_HELPERDIR - Override default HELPERDIR value ($LIBEXECDIR/nhc)
    • NHC_CFG_LIBEXECDIR - Override default LIBEXECDIR value (OS-dependent)
    • NHC_CFG_LOGDIR - Override default LOGDIR value ($STATEDIR/log)
    • NHC_CFG_RUNDIR - Override default RUNDIR value ($STATEDIR/run/nhc)
    • NHC_CFG_STATEDIR - Override default STATEDIR value (OS-dependent)
    • NHC_CFG_SYSCONFIGDIR - Override default SYSCONFIGDIR value (OS-dependent)
• check_cmd_output() and check_file_contents() now support conditionals, additional expressions that can be used to fine-tune or supplement match strings, along with a whole slew of other new options. See README file for details.

Fixes/Improvements

• Resource consumption checks now differentiate between system and non-system users.
• Improved handling for Slurm node states and transitions
• Logs successful completion, not just failure
• More informative and customizable syslog() labeling and tagging
• Hardware checks reading from procfs/sysfs files have been refactored to allow immensely greater flexibility as well as to address performance issues on some systems (e.g., many-core Knights Landing CPUs). Data sources are now completely user-configurable and can be cached copies, files in other locations, or even subcommands/pipelines.
• Node online/offline helpers are now invoked in a subshell for greater data integrity protection.
• BREAKING CHANGE: check_file_stat() now uses 2 distinct option sets that examine the file's mode. In previous versions, -M/-m looked solely at the permissions bits (i.e., mode & 00007777), whereas in NHC 1.5, this is done by the -P/-p "permissions" options. The -M/-m "mode" options now look at the file type bits (mode & 00170000).
• Options to check_file_test() can now be bundled.
• When attempting to automatically fix a missing mount point, NHC will now accurately report the success or failure of the mkdir attempt.
• After loading checks from the config file, if the resulting check list is empty, a warning is now logged rather than nhc just exiting silently.
• Special handling of Bash exit codes 126 and 127 has been added to help point admins to the most likely causes.
• When running under Slurm, the online helper will no longer attempt to resume nodes in the maint state; slurmctld rejects this as an invalid state change.
• check_file_contents() now uses the same backend processing/matching functionality that check_cmd_output() uses, enabling use of all the same options and arguments.

Merged PRs

• Add csc_nvidia_smi.nhc to Makefile to get into RPM by @treydock in #114
• Numerous improvements to Slurm reboot handling and planned node state by @treydock in #115
• Fix unsafe string array conversion for passwd file parsing by @lebonez in #117
• lbnl_hw: Fixes/speedups for procfs file reads by @mej in #121
• nhc: Use subshell syntax for offline/online ops by @mej in #131
• lbnl-nhc.spec.in: Allow for normal user access by @mej in #133
• nhc: Refactor watchdog timer code for reuse by @mej in #132
• nhc: Add concurrency checking via PID file by @mej in #140
• nhc: Print check numbers in verbose mode (#124) by @mej in #142
• lbnl_cmd.nhc: Remove line numbers from dmesg check by @mej in #144
• Resuming node from state maint is an invalid node state transition by @papamoose in #176

New Contributors

• @lebonez made their first contribution in #117
• @papamoose made their first contribution in #176

The Gory Details

• Full Release Notes: RELEASE_NOTES.md
• Full Changelog: 1.4.3...1.5