Setup hooks don't work for CompileOnlyRegressionTest #1162
Description
The setup hooks are not called at all before/after the setup phase when a test is CompileOnlyRegressionTest. The reason is that the setup() function is overwritten in the CompileOnlyRegressionTest and the decorator is not inherited.
Similarly in the RunOnlyRegressionTest the hooks for the run phase are called but not where we expect it. It is called before/after the call to the base class of run inside RunOnlyRegressionTest.run().
Here is a simple example. When the test is RegressionTest.
import reframe as rfm
import reframe.utility.sanity as sn
@rfm.simple_test
class Example1Test(rfm.RegressionTest):
def __init__(self):
self.descr = 'Simple matrix-vector multiplication example'
self.valid_systems = ['*']
self.valid_prog_environs = ['*']
self.sourcepath = 'example_matrix_vector_multiplication.c'
self.executable_opts = ['1024', '100']
self.sanity_patterns = sn.assert_found(
r'time for single matrix vector multiplication', self.stdout)
self.maintainers = ['you-can-type-your-email-here']
self.tags = {'tutorial'}
@rfm.run_before('setup')
def test_hook(self):
assert(0 == 1)
Output is failing (as it should) because of the hook:
Command line: ./bin/reframe -c tutorial/example1.py --skip-sanity -r
Reframe version: 3.0-dev1
Launched by user: ekoutsaniti
Launched on host: dhcp-133-245.cscs.ch
Reframe paths
=============
Check prefix :
Check search path : 'tutorial/example1.py'
Stage dir prefix : /Users/ekoutsaniti/repos/reframe-fork/stage/
Output dir prefix : /Users/ekoutsaniti/repos/reframe-fork/output/
Perf. logging prefix : /Users/ekoutsaniti/repos/reframe-fork/perflogs
[==========] Running 1 check(s)
[==========] Started on Fri Feb 14 08:39:30 2020
[----------] started processing Example1Test (Simple matrix-vector multiplication example)
[ RUN ] Example1Test on generic:login using builtin-gcc
[ FAIL ] Example1Test
[ FAILED ] Ran 1 test case(s) from 1 check(s) (1 failure(s))
[==========] Finished on Fri Feb 14 08:39:30 2020
==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for Example1Test
* System partition: None
* Environment: None
* Stage directory: None
* Node list: <None>
* Job type: batch job (id=-1)
* Maintainers: ['you-can-type-your-email-here']
* Failing phase: setup
* Reason: unexpected error:
Traceback (most recent call last):
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/__init__.py", line 163, in _safe_call
return fn(*args, **kwargs)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/core/pipeline.py", line 94, in _fn
h(obj)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/core/decorators.py", line 180, in _fn
func(*args, **kwargs)
File "/Users/ekoutsaniti/repos/reframe-fork/tutorial/example1.py", line 19, in test_hook
assert(0 == 1)
AssertionError
------------------------------------------------------------------------------
./bin/reframe: unexpected error:
Traceback (most recent call last):
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/cli.py", line 608, in main
runner.runall(testcases)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/__init__.py", line 284, in runall
self._runall(testcases)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/__init__.py", line 336, in _runall
self._policy.runcase(t)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/policies.py", line 268, in runcase
if not self._setup_task(task):
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/policies.py", line 237, in _setup_task
sched_options=self.sched_options)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/__init__.py", line 172, in setup
self._safe_call(self.check.setup, *args, **kwargs)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/frontend/executors/__init__.py", line 163, in _safe_call
return fn(*args, **kwargs)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/core/pipeline.py", line 94, in _fn
h(obj)
File "/Users/ekoutsaniti/repos/reframe-fork/reframe/core/decorators.py", line 180, in _fn
func(*args, **kwargs)
File "/Users/ekoutsaniti/repos/reframe-fork/tutorial/example1.py", line 19, in test_hook
assert(0 == 1)
AssertionError
But if it is defined as CompileOnly and we skip the sanity it passes:
import reframe as rfm
import reframe.utility.sanity as sn
@rfm.simple_test
class Example1Test(rfm.CompileOnlyRegressionTest):
def __init__(self):
self.descr = 'Simple matrix-vector multiplication example'
self.valid_systems = ['*']
self.valid_prog_environs = ['*']
self.sourcepath = 'example_matrix_vector_multiplication.c'
self.executable_opts = ['1024', '100']
self.sanity_patterns = sn.assert_found(
r'time for single matrix vector multiplication', self.stdout)
self.maintainers = ['you-can-type-your-email-here']
self.tags = {'tutorial'}
@rfm.run_before('setup')
def test_hook(self):
assert(0 == 1)
Output:
Command line: ./bin/reframe -c tutorial/example1.py --skip-sanity -r
Reframe version: 3.0-dev1
Launched by user: ekoutsaniti
Launched on host: dhcp-133-245.cscs.ch
Reframe paths
=============
Check prefix :
Check search path : 'tutorial/example1.py'
Stage dir prefix : /Users/ekoutsaniti/repos/reframe-fork/stage/
Output dir prefix : /Users/ekoutsaniti/repos/reframe-fork/output/
Perf. logging prefix : /Users/ekoutsaniti/repos/reframe-fork/perflogs
[==========] Running 1 check(s)
[==========] Started on Fri Feb 14 08:40:20 2020
[----------] started processing Example1Test (Simple matrix-vector multiplication example)
[ RUN ] Example1Test on generic:login using builtin-gcc
[----------] finished processing Example1Test (Simple matrix-vector multiplication example)
[----------] waiting for spawned checks to finish
[ OK ] Example1Test on generic:login using builtin-gcc
[----------] all spawned checks have finished
[ PASSED ] Ran 1 test case(s) from 1 check(s) (0 failure(s))
[==========] Finished on Fri Feb 14 08:40:20 2020