The executor property can support regular expression to search for compatible executors, this can be used if you want to run a test across multiple executors. In buildtest, we use re.fullmatch with the input pattern defined by executor property against a list of available executors defined in configuration file. You can retrieve a list of executors by running buildtest config executors list.
In example below we will run this test on generic.local.bash and generic.local.sh executor based on the regular expression.
../tutorials/multi_executors/executor_regex_script.yml
If we build this test, notice that there are two tests, one for each executor.
buildtest build -b tutorials/multi_executors/executor_regex_script.yml
buildtest build -b tutorials/multi_executors/executor_regex_script.yml
Note
This feature is in active development
Note
This feature is compatible with type: script and type: spack.
The executors property can be used to define executor specific configuration for each test, currently this field can be used with vars <variables>, env <environment_variables> , scheduler directives: sbatch, bsub, pbs, cobalt and cray burst buffer/data warp <cray_burstbuffer_datawarp>. The executors field is a JSON object that expects name of executor followed by property set per executor. In this next example, we define variables X, Y and environment SHELL based on executors generic.local.sh and generic.local.bash.
../tutorials/multi_executors/executors_var_env_declaration.yml
Let's build this test.
buildtest build -b tutorials/multi_executors/executors_var_env_declaration.yml
buildtest build -b tutorials/multi_executors/executors_var_env_declaration.yml
Now let's look at the generated content of the test as follows. We will see that buildtest will set X=1, Y=3 and SHELL=bash for generic.local.bash and X=2, Y=4 and SHELL=sh for generic.local.sh
buildtest inspect query -t executors_vars_env_declaration/
buildtest inspect query -t executors_vars_env_declaration/
We can also define scheduler directives based on executor type, in this example we define sbatch property per executor type. Note that sbatch property in the executors section will override the sbatch property defined in the top-level file otherwise it will use the default.
../tutorials/multi_executors/executor_scheduler.yml
buildtest build -b tutorials/multi_executors/executor_scheduler.yml
buildtest build -b tutorials/multi_executors/executor_scheduler.yml
If we inspect this test, we will see each each test have different #SBATCH directives for each test based on the sbatch property defined in the executors field.
buildtest inspect query -t executors_sbatch_declaration/
buildtest inspect query -t executors_sbatch_declaration/
You can also define BB and DW directives in the executors field to override cray burst buffer and data warp settings per executor. buildtest will use the fields BB and DW and insert the #BB and #DW directives in the job script. For more details see cray_burstbuffer_datawarp.
../tutorials/burstbuffer_datawarp_executors.yml
The status <status> and metrics <metrics> field are supported in executors which can be defined within the named executor. In this next example, we will define executor generic.local.bash to match for returncode 0 or 2 while second test will use executor generic.local.sh to match returncode of 1.
../tutorials/multi_executors/status_by_executors.yml
Now let's run this test and we will see the test using executor generic.local.sh will fail because we have a returncode mismatch even though both tests got a 0 returncode as its actual value.
buildtest build -b tutorials/multi_executors/status_by_executors.yml
buildtest build -b tutorials/multi_executors/status_by_executors.yml