Is a validation stage needed, to make sure a test contains a set of required parameters?

[kraushm]

During the creation of the Fieldextra performance test, Reframe submitted all 3 tests (2x accuracy + 1x perf) through SLURM, although the class FieldextraPerformance(FieldextraCheck) (and the base class FieldextraCheck) did not contain any sanity patterns. (see output below)

Command line: ./reframe.py -C config/cscs.py -c cscs-checks/mch/fieldextra_check.py --run --keep-stage-files --prefix=/scratch/kraushm/.
Reframe version: 2.17-dev0
Launched by user: kraushm
Launched on host: keschln-0003
Reframe paths
=============
    Check prefix      :
    Check search path : 'cscs-checks/mch/fieldextra_check.py'
    Stage dir prefix     : /scratch/kraushm/stage/
    Output dir prefix    : /scratch/kraushm/output/
    Perf. logging prefix : /scratch/kraushm/perflogs
[==========] Running 3 check(s)
[==========] Started on Fri Feb 22 10:43:33 2019[----------] started processing FieldextraAccuracy_opt (Fieldextra validation test (accuracy); MCH)
[ RUN      ] FieldextraAccuracy_opt on kesch:cn using PrgEnv-gnu-nompi
[       OK ] FieldextraAccuracy_opt on kesch:cn using PrgEnv-gnu-nompi
[----------] finished processing FieldextraAccuracy_opt (Fieldextra validation test (accuracy); MCH)[----------] started processing FieldextraAccuracy_opt_omp (Fieldextra validation test (accuracy); MCH)
[ RUN      ] FieldextraAccuracy_opt_omp on kesch:cn using PrgEnv-gnu-nompi
[       OK ] FieldextraAccuracy_opt_omp on kesch:cn using PrgEnv-gnu-nompi
[----------] finished processing FieldextraAccuracy_opt_omp (Fieldextra validation test (accuracy); MCH)[----------] started processing FieldextraPerformance (Fieldextra validation test (performance); MCH)
[ RUN      ] FieldextraPerformance on kesch:cn using PrgEnv-gnu-nompi
[     FAIL ] FieldextraPerformance on kesch:cn using PrgEnv-gnu-nompi
[----------] finished processing FieldextraPerformance (Fieldextra validation test (performance); MCH)[  FAILED  ] Ran 3 test case(s) from 3 check(s) (1 failure(s))
[==========] Finished on Fri Feb 22 10:53:11 2019==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for FieldextraPerformance
  * System partition: kesch:cn
  * Environment: PrgEnv-gnu-nompi
  * Stage directory: /scratch/kraushm/stage/kesch/cn/PrgEnv-gnu-nompi/FieldextraPerformance
  * Job type: batch job (id=1009420)
  * Maintainers: ['Mkr']
  * Failing phase: sanity
  * Reason: sanity error: sanity_patterns not set
------------------------------------------------------------------------------

The error message is correct, but the test wasted compute resources, since it does not check anything. Should there be an additional stage in the pipeline, which would filter out "incomplete" tests, e. g. by checking for certain (required) keywords/parameters?

[vkarak]

In the validation for a test we should check its ability to be deep copied as specified in #55.