Harness crash and stack trace dump

[hagertnl]

This rgt_test_input.ini file caused the harness to dump a stack trace. The file may not be correct, but we should catch this error better.

[DEFAULT]
rocprim_build_version = 5.7.1

[Replacements]
# Harness-required parameters:
job_name = rocprim_nodecheck
nodes = <obtain_from_environment>
walltime = 20
batch_filename = run_rocprim_nodecheck.sh
executable_path = rocprim/rocprim-prebuilt-%(rocprim_build_version)s.tar.gz
build_cmd = ./build.sh %(executable_path)s
check_cmd = ./check.sh
report_cmd = /usr/bin/echo
resubmit = 1

# Harness-populated environment variables that we need access to
# RGT_MACHINE_NAME is needed for locating executable sometimes
machine_name = <obtain_from_environment>

# Non-harness replacements:
prebuilt_path = /sw/acceptance/amdscreen
# 1: one srun per node. 0: for one srun command covering all nodes
launch_separate = 0
# Notes:
#   -Please keep the -w ${nodename} in, as this is the variable name in the for-loop
#   -Use /tmp/binaryname for your executable, since the executable is `sbcast` out, with it's libraries
launch_command = srun -c56 -t25 --gpus-per-task=8 -N${SLURM_NNODES} -n${SLURM_NNODES} --ntasks-per-node=1 --kill-on-bad-exit=0 --no-kill --mpi=none /tmp/rocprim-wrapper.sh 1

[EnvVars]
# Env-vars are needed for the build script to operate
PREBUILT_PATH = %(prebuilt_path)s

Stack:

Creating machine borg: Type = linux_x86_64 ; Scheduler = slurm ; Job launcher = srun
Creating scheduler
Creating jobLauncher
Traceback (most recent call last):
  File "/sw/acceptance/olcf-test-harness/harness/bin/test_harness_driver.py", line 539, in <module>
    exit_code = test_harness_driver()
  File "/sw/acceptance/olcf-test-harness/harness/bin/test_harness_driver.py", line 499, in test_harness_driver
    Vargs.separate_build_stdio)
  File "/sw/acceptance/olcf-test-harness/harness/bin/test_harness_driver.py", line 168, in auto_generated_scripts
    mymachine = MachineFactory.create_machine(harness_config, apptest, separate_build_stdio=separate_build_stdio)
  File "/sw/acceptance/olcf-test-harness/harness/machine_types/machine_factory.py", line 93, in create_machine
    apptest=app_subtest)
  File "/sw/acceptance/olcf-test-harness/harness/machine_types/linux_x86_64.py", line 47, in __init__
    self._rgt_test.read_input_file()
  File "/sw/acceptance/olcf-test-harness/harness/machine_types/rgt_test.py", line 433, in read_input_file
    self._read_rgt_input_ini()
  File "/sw/acceptance/olcf-test-harness/harness/machine_types/rgt_test.py", line 510, in _read_rgt_input_ini
    self.test_environment = env_vars
  File "/sw/acceptance/olcf-test-harness/harness/machine_types/rgt_test.py", line 271, in test_environment
    self.__environ.update(envvars_view)
  File "/usr/lib64/python3.6/configparser.py", line 1234, in __getitem__
    return self._parser.get(self._name, key)
  File "/usr/lib64/python3.6/configparser.py", line 800, in get
    d)
  File "/usr/lib64/python3.6/configparser.py", line 394, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "/usr/lib64/python3.6/configparser.py", line 437, in _interpolate_some
    section, map, depth + 1)
  File "/usr/lib64/python3.6/configparser.py", line 437, in _interpolate_some
    section, map, depth + 1)
  File "/usr/lib64/python3.6/configparser.py", line 437, in _interpolate_some
    section, map, depth + 1)
  [Previous line repeated 7 more times]
  File "/usr/lib64/python3.6/configparser.py", line 409, in _interpolate_some
    raise InterpolationDepthError(option, section, rawval)
configparser.InterpolationDepthError: Recursion limit exceeded in value substitution: option 'prebuilt_path' in section 'EnvVars' contains an interpolation key which cannot be substituted in 10 steps. Raw value: '%(prebuilt_path)s'
In function subtest._start_test we have a critical error.
The command 'test_harness_driver.py -r -l borg_test/hagertnl@2024-05-07T23:20:39.12 --loglevel WARNING' has exited with a failure.
The exit return value is 1
.
Application every-node future exception:
Command failed: test_harness_driver.py -r -l borg_test/hagertnl@2024-05-07T23:20:39.12 --loglevel WARNING
Launched 0 tests, skipped 1 tests.

[hagertnl]

This was caused by trying to use a non-DEFAULT section variable outside of the section that it's defined in. We might want to look into enabling extended interpolation syntax, which gives the developer more granularity over their input file variables: https://docs.python.org/3/library/configparser.html#configparser.ExtendedInterpolation

[hagertnl]

Tightly coupled to #135