Print summary table and run options (optionally) for failures #1161
Description
@vkarak Let us discuss the detail of failure summary table here before I pull request.
For summary table, here is an example:
SUMMARY OF FAILURES
------------------------------------------------------------------------------
[ .. skipped .. ]
------------------------------------------------------------------------------
Total number of tests: 669
Total number of failures: 39
Phase # Description
-------- ------ --------------------------------------------------------------
run 32 Failed to launch jobs
compile 6 Failed to compile the source code in the current environment
sanity 1 Failed in sanity checking
More precise description will be helpful for troubleshooting.
In my testing routine, I often convert the detail of failures to command-line options so I can rerun failed tests after fixing them. It is kind of like --rerun-failed in CTest. It is not common for others so I would like to make it optional:
$ reframe.py -t apps --rerun-failed -r
[ .. skipped .. ]
Total number of tests: 669
Total number of failures: 39
Phase # Description
-------- ------ --------------------------------------------------------------
run 32 Failed to launch jobs
no module: -n openmpi_default_pgi -p pgi-default --system owens:login
no module: -n openmpi_default_pgi -p pgi-default --system owens:pbs
no module: -n openmpi_test_mpi_placement -p pgi-default --system owens:pbs
no module: -n openmpi_module_path -p pgi-default --system owens:login
no module: -n openmpi_module_path -p pgi-default --system owens:pbs
run: -n darshan_mpi_test -p gnu-mvapich2 --system owens:pbs
[ .. skipped .. ]
Prepend a tag to the command-line options of a failure so one can decide if some failures can be skipped.
The custom changes can be found here:
ZQyou@8d969c4#diff-4cc7123431b59755e1d77edbb2347fef (statistics.py)
ZQyou@8d969c4#diff-48e18b1d45e8fe87d74f3687c93fad73 (cli.py)