Testing an installation

[eschnett]

When installing software, I like to run a quick test that the installation works fine. This is nothing more than building and running a trivial program that outputs the version numbers. Often, this catches problems with install paths or library paths. It would be good to support this as a post-install step.

[tgamblin]

👍 This is planned but not yet on the roadmap. Some teams at LLNL have also asked for configure and install to be separated, so that will likely happen as well.

[davydden]

👍 for a test block in each package as a post-install step invoked by spack test <package>. Here are a couple of examples from Homebrew: deal.ii, mumps, petsc.

Also many packages are missing make test runs or equivalent, which normally do not take more than a minute or so but are good to check that things are working as expected before doing production runs.

[adamjstewart]

I've definitely run into some make test runs that take significantly longer than the configure or make steps, so we may not want them enabled by default. Personally, I would love to run make test so I can sleep well at night. If we separate configure and install, then we could easily add a test() function. We could then add a -t/--test option to make install that runs test().

One thing to consider would be something like GCC, which requires a few additional dependencies to properly run the test suite. There has already been talk of separating build dependencies and runtime dependencies. This may require us to add test dependencies.

[tgamblin]

@davydden: I would really like to add testing capability. One challenge is that Spack is designed for big machines, and the command to actually run stuff (esp. on BG/Q, XC40, etc.) can be very different from site to site. I would want to add a generic, auto-detected msub/qsub/srun command that could be used in Spack tests.

[davydden]

@tgamblin I was thinking about the following HPC usage case: create a job for a single node which does nothing for a couple of hours; ssh to the node; run spack to install everything. Since it's on the node, one will be able to use mpiexec or whatever name it has locally, which is not always the case on the front-end.

[alalazo]

@tgamblin we would also be interested in scheduler support within spack. If it may be of interest this is the only package I found that seems to go in the direction of supporting different schedulers. Didn't have much time to play with it though...

[adamjstewart]

How do people feel about including make("test") in packages?

[mathstuf]

Depends on how reliable the test suites are. There should probably be a --no-test argument to spack install to suppress testing so that test suites which take a long time (I'm thinking at least VTK and ParaView here, but they're not the only ones) can be skipped.

[davydden]

i would not have a separate option --no-test, instead all the tests which are reasonable fast should be done unconditionally. Whereas tests which are long should be done (if possible) by something like spack test <package>.

The rationale is that --no-test does not change the variant of the package (like +openmp, +mpi, +shared, etc) so i would avoid those.

[mathstuf]

Well, there's the problem that the default is to not keep stage directories around, so a spack test would require rebuilding the package from scratch in the "common" case.

[alalazo]

I think we can distinguish between install time tests (make test or similar things) and post-install tests (like the small executable that @davydden provided for openblas). For the former I agree with @mathstuf in that spack install --no-tests ... is needed, for the latter I think we should be able to execute them after the installation like @davydden proposes.

[alalazo]

A couple random thoughts on this:

• --run-tests install argument and petsc fixes #1169 added a switch to run tests at install time
• do_install : allow for an arbitrary number of phases #1186 might provide a modular solution to attach custom tests to packages

[davydden]

i think essentially when/if #1186 is merged, what's missing is spack test <package> which would run a post-install step install_test(), if it is defined in a package.

[alalazo]

@eschnett Is there something more that needs to be done? Or are you fine with :

spack install --run-tests ...

if appropriate functions have been defined in the package (like in hdf5 fo instance)?

[alalazo]

@eschnett I am going to close this because I think it has been fixed already. Please reopen if still necessary.