More informative messages when checks are undetected

[shoshijak]

I'm trying out reframe by running a simple hello world, like this:

@rfm.simple_test
class HelloWorldTest(rfm.RegressionTest):
    def __init__(self):
        super().__init__()
        self.descr = 'C++ Hello World Test'
        self.name="hello-world"
        self.valid_systems = ['daint:mc']
        self.valig_prog_environs = ['*']
        self.sourcesdir = os.path.join(self.prefix, '..')
        self.build_system = 'SingleSource'
        self.sanity_patterns = sn.assert_found(r'Hello world!', self.stdout)
        self.tags = {'hello-world'}

When I run reframe however, no checks are detected:

Command line: /apps/common/UES/jenkins/SLES12/easybuild/software/reframe/2.17/bin/reframe -C /users/alicej/reframe/settings.py -c reframe/dummy.py -l
Reframe version: 2.17
Launched by user: alicej
Launched on host: daint103
Reframe paths
=============
    Check prefix      :
    Check search path : 'reframe/dummy.py'
    Stage dir prefix     : /users/alicej/dbcsr/stage/
    Output dir prefix    : /users/alicej/dbcsr/output/
    Perf. logging prefix : /users/alicej/dbcsr/perflogs
List of matched checks
======================
Found 0 check(s).

It took me a while to figure out that this happened because I had simply misspelled "valid":

                 |
                 V
        self.valig_prog_environs = ['*']

Since no information was printed on the reason why no checks were found, finding the spelling error was not straightforward.

Just as a suggestion for user-friendliness, would it be possible, for example:

• to add a warning or message if non-supported attributes are added to the class (e.g. in this case valig_prog_environs),
• or to add a warning message if a test was instantiated, but no valid programming environments were found/specified for it, as is also the case here?

[vkarak]

Yeah, we've gone through it before and it's a bit annoying. Good that you opened an issue for that! Regarding your suggestions:

to add a warning or message if non-supported attributes are added to the class (e.g. in this case valig_prog_environs),

This is not possible, cos there's no such thing as "non-supported" attributes. We want users to define their own attributes.

or to add a warning message if a test was instantiated, but no valid programming environments were found/specified for it, as is also the case here?

That's probably the way to go. We should issue a warning if valid_prog_environs is empty or even better change its default to None so that we can still allow users to explicitly set it to an empty list, if for example they want to "disable" a test. We can do the same also for valid_systems as well.

[shoshijak]

That's probably the way to go. We should issue a warning if valid_prog_environs is empty or even better change its default to None so that we can still allow users to explicitly set it to an empty list, if for example they want to "disable" a test. We can do the same also for valid_systems as well.

Yes, this sounds like a good solution :)

[vkarak]

This is related to #701.