Adding an option for a pre-workload script that is run before the IO exerciser

[harriscr]

This PR adds the "pre_workload_script" to a workload specified in the configuration yaml file. It is specific to the workload, and has no effect outside of the workloads section of the yaml file.

For more information see the workloads documentation in CBT.

It is the user responsibility to make sure that the script is available at the same path for all clients used in the test run.
The script must exit with a return code that matches the OS schema i.e. for Linux return code 0 is good, and anything else is considered bad.

The script is run on every client using the standard common.pdsh() method in CBT.

This has been tested locally with a script that deletes and recreates volumes for a workload to run on before the fio IO is run. See the below output snippet gtom the cbt log file that shows the script being scheduled and run:

22:05:48 - DEBUG    - cbt      - Scheduling script /cbt.lee/tools/setup_cluster/mkdelvols.cbt to run
22:05:48 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST /cbt.lee/tools/setup_cluster/mkdelvols.cbt
22:06:10 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-0
22:06:10 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.0
22:06:10 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-1
22:06:10 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/output.1
22:06:40 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/collectl
22:06:40 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST collectl -c 18 -sCD -i 10 -P -oz -F0 --rawtoo --sep ";" -f /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-002/numjobs-001/collectl
22:07:41 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST pkill -SIGINT -f collectl
22:07:41 - WARNING  - cbt      - The total iodepth requested: 4 is less than 1 per volume (8)
22:07:41 - WARNING  - cbt      - Number of volumes per client will be reduced from 8 to 4
22:07:41 - DEBUG    - cbt      - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w root@HOST mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001
22:07:41 - DEBUG    - cbt      - Scheduling script /cbt.lee/tools/setup_cluster/mkdelvols.cbt to run
22:07:41 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST /cbt.lee/tools/setup_cluster/mkdelvols.cbt
22:07:59 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-0
22:07:59 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.0
22:07:59 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-1
22:07:59 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.1
22:07:59 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-2
22:07:59 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-2 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.2 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.2
22:07:59 - DEBUG    - cbt      - Using rbdname cbt-librbdfio-`hostname -f`-3
22:07:59 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-3 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.3 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.3 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.3 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/output.3
22:08:30 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/collectl
22:08:30 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST collectl -c 18 -sCD -i 10 -P -oz -F0 --rawtoo --sep ";" -f /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-004/numjobs-001/collectl
22:09:30 - DEBUG    - cbt      - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w root@HOST pkill -SIGINT -f collectl
22:09:30 - DEBUG    - cbt      - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w root@HOST mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-008/numjobs-001

I will update teuthology runs here once they have completed

[lee-j-sanders]

I've been using this feature for a few days now, and it works very well.
Looks good to me. Approved.