Added support for multiple user-specified applications to be run in integration tests

[bieryAtFnal]

Description

In response to a request from Emir for the ability to start a drunc-process-manager instance in addition to the drunc-unified-shell instance in an integration test, functionality has been added to the integrationtest infrastructure to support user-specified applications.

This new functionality is controlled by a new special variable that developers can include in their integtests. This new special variable is described in a new document in this repository. This document will eventually describe all of the special variables that can/should be used in integtests, and it can be viewed here.

These changes have been made in a largely backward-compatible way, so most existing regression/integration tests will not need to be changed. However, the status information that is passed back to the integtest/pytest "tests" now has information about all of the control applications that were run in the DAQ session, so integtests that inspect those results will now need to loop over the list of results instead of simply looking at a single status object.

The console output that users will see when they run an integtest with these changes is largely unchanged, but at a verbosity level of 4 or greater, users will now see information about the processes that are started and the commands that are sent to them.

These changes are correlated with the ones in DUNE-DAQ/drunc#997. The changes in the drunc repo include a new integtest that demonstrates multiple user-specified applications that are run in the DAQ session for the test.

One of the reasons for creating this PR in "draft" mode is that there are still some changes needed in the drunc/process_manager_test.py in order to get it to work.

Here are suggested commands for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260805_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/multi_ctrl_proc_support

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/multi_ctrl_proc_support
git clone https://github.com/DUNE-DAQ/drunc.git -b kbiery/multi_ctrl_proc_support
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh --verb 4 -r drunc

echo ""
echo -e "\U1F535 \U2705 Note that the new basic_multiapp_test integtest worked, but the process_manager_test did not. \U2705 \U1F535"
echo ""
echo ""

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Unit tests pass (e.g. dbt-build --unittest)
• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas

[emmuhamm]

Going through the docs (very complete and well written btw, thanks!) and a minor suggestion:

Suggested change

	```
	```python

As this bit of code is Python, flavoring this codeblock with the programming language will allow GitHub to do the linting so it looks more legible.

Example

# The commands to run in dunerc and the process manager shell
dunerc_commands_1 = (
    "boot conf start --run-number 101 wait 1 enable-triggers wait ".split()
    + [str(run_duration)] + ["disable-triggers"]
)
dunerc_commands_2 = (
    "drain-dataflow stop-trigger-sources stop wait 2 scrap terminate".split()
)
pmshell_command = ["ps"]

vs

# The commands to run in dunerc and the process manager shell
dunerc_commands_1 = (
    "boot conf start --run-number 101 wait 1 enable-triggers wait ".split()
    + [str(run_duration)] + ["disable-triggers"]
)
dunerc_commands_2 = (
    "drain-dataflow stop-trigger-sources stop wait 2 scrap terminate".split()
)
pmshell_command = ["ps"]

[emmuhamm]

Can you say a few words in the docs about what DAQCommandSet does and what CommandWaitParaameters refer to and their purpose? the command set looks quite self explanatory from the example but I'm still not sure what the CommandWaitParameters are for, especially the difference between ECHO and TIME