Spack Stand Alone Smoke Testing Support
Stand-alone (or smoke tests) are intended to be quick checks of the installation of a package -- weeks to months later -- to determine whether more extensive regression testing should be performed and to provide confidence that the installation still works as the system evolves. These tests will ideally check the core functionality of the package.
Refer to the Spack documentation on stand-alone tests for additional information.
Stand-alone test support was added in PR 15702 and integrated into Spack as an experimental feature in v0.16. Preliminary capabilities include the ability to:
- define multi-part tests;
- copy files and directories from staged source during the installation process;
- run arbitrary executables (including compilation and linking);
- add arbitrary files for use in the tests; and
- run smoke tests on an arbitrary number of packages under a single name (or alias).
Spack tracks and logs the output of multi-part tests during execution when the self.run_test() method is used for test parts.
usage: spack test [-h] SUBCOMMAND ...
run spack's tests for an install
positional arguments:
SUBCOMMAND
run Run tests for the specified installed packages.
list List all installed packages with available tests.
find Find tests that are running or have available results.
status Get the current status for the specified Spack test suite(s).
results Get the results from Spack test suite(s) (default all).
remove Remove results from Spack test suite(s) (default all).
optional arguments:
-h, --help show this help message and exit
The table below reflects early smoke tests for a subset of packages intended to illustrate preliminary smoke (or install) tests for packages. Additional packages have been included when their smoke tests add a new twist to the process. You can get a list of packages that have at least preliminary tests using the spack test list command:
usage: spack test list [-ha] [tag [tag ...]]
List installed packages with available tests.
positional arguments:
tag limit packages to those with all listed tags
optional arguments:
-a, --all list all packages with tests (not just installed)
-h, --help show this help message and exit
In some cases, tests may only be inherited from parent and or virtual packages (e.g., PythonPackage, mpi providers).
The meaning of each column in the table below is:
- Package is the name of the built-in package
- Version Checks has a Y (for yes) if the smoke test includes version checks for one or more installed binary, otherwise N (for no)
- Copies From Source has a Y if the package copies files and or directories during the install process, otherwise N
- Compiles Source has a Y if the smoke tests compile source file(s) before running smoke tests, otherwise N
- Adds test has a Y if the package has a test subdirectory containing one or more smoke test file, otherwise N
- Notes provide optional information about the package smoke tests
| Package | Version Checks | Copies From Source | Compiles Source | Adds test | Notes |
|---|---|---|---|---|---|
| amrex | N | Y | Y | N | Uses make to build and run the tests |
| arborx | N | Y | Y | N | Uses cmake, make, and ctest for building and running the tests |
| berkeley-db | Y | N | N | N | |
| binutils | Y | N | N | N | |
| biobambam2 | N | Y | N | N | Copies three bash scripts; uses Spack's filter_file to modify the paths in one script to work as a smoke test |
| c | N | N | Y | Y | Adds one test source file |
| cmake | Y | N | N | N | |
| cxx | N | N | Y | Y | Adds four test source files |
| emacs | Y | N | N | N | |
| fortran | N | N | Y | Y | Adds two test source files |
| hdf | Y | Y | N | Y | Copies test files; runs sequence of binaries from the test files; adds expected test output |
| hdf5 | Y | N | N | Y | Adds simple hdf5 file and expected dump output; runs existing check_install |
| heffte | N | Y | Y | N | uses make to build and run the tests |
| libxml2 | Y | N | N | Y | Adds DTD and XML files used in tests of installed binaries; runs existing import_module_test |
| m4 | Y | N | N | Y | Adds example m4 and expected output files |
| mpi | N | N | Y | Y | (VIRTUAL) Adds, compiles, and runs (on a single processor) two test source files |
| openmpi | Y | Y | Y | N | Copies, builds (via make), and runs source examples; also runs mpi tests |
| parallel-netcdf | N | Y | Y | N | Copies and compiles example; runs using any/all available from mpirun, mpiexec, and srun; also cleans up |
| patchelf | Y | N | N | Y | Adds and uses example |
| perl | Y | N | N | N | Hard codes and uses an example |
| py-accimage | N | N | N | N | TBD: Is the test method meant to be a smoke test? |
| py-vermin | N | N | N | N | Runs make test; TBD: Is the test method meant to be a smoke test?
|
| py-warpx | N | N | N | N | Uses module import tests inherited from the python package |
| python | N | N | N | N | Uses hard-coded examples |
| raja | N | Y | N | N | Copies build examples; runs subset of examples |
| sqlite | Y | N | N | Y | Adds and runs binaries against simple example database |
| umpire | N | Y | N | N | Copies build tests; runs subset of copied benchmark, cookbook, example, plots, tools, and tutorials |
| warpx | N | Y | N | N | Runs their app (only if built) |