Note
Please see tutorial_setup before you proceed with this section
The E4S Testsuite is a collection of lightweight tests assembled for over 100 software products provided by Extreme Scale Scientific Software Stack (E4S) deployments. E4S provides a spack based software stack. Not all spack packages provide internal tests and package-provided tests are static and potentially difficult to modify for use at facility systems. The E4S Testsuite provides a flexible, alternative testing framework for verifying spack-deployed software.
The E4S Testsuite operates on a hierarchy of shell scripts with each test containing its own setup.sh, clean.sh, compile.sh and run.sh scripts. The top level driver script sets up the test environment based on a configuration file which sets compilers, mpi run commands and other environment specific parameters. It then iterates through the selected tests and generates output consisting of log files for individual test runs and a summary listing of each test’s success or failure. The README provides more detail on its structure and usage.
The provided tests typically include source code, build, and input files for quick-running example applications. Some import test code from the installed product's directory or invoke spack internal tests (spack test run) directly.
E4S Testsuite provides and maintains pre-constructed tests for a number of products deployed at NERSC. It often makes sense to invoke these tests rather than develop new ones. Buildtest buildspecs that make use of the E4S Testsuite follow the pattern of loading the relevant spack modules, acquiring the testsuite and executing the test driver script on the selected test.
Shown below is an example buildspec running E4S Testsuite for testing the spack package mpich. In order to test mpich, we need to install mpich into our container environment by running spack install mpich prior to testing the package. We will clone E4S Testsuite and invoke the driver script test-all.sh for running the test.
../../examples/spack/e4s_testsuite_mpich.yml
We can see an example of the E4S Testsuite in action by running the following command.
buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml
cat buildtest_tutorial_examples/spack/build/e4s_testsuite_mpich.txt
E4S Testsuite prints full test logs to the standard error instead of stdout so we must use buildtest inspect query -e to view the full test results. Let's view the test output by running the following:
buildtest inspect query -o -e -t mpich_e4s_testsuite
cat buildtest_tutorial_examples/spack/inspect/e4s_testsuite_mpich.txt