[feat] Improve handling of user programming errors

docs/config_reference.rst

@@ -1280,6 +1280,10 @@ General Configuration
 
    Ignore test name conflicts when loading tests.
 
+   .. deprecated:: 3.8.0
+      This option will be removed in a future version.
+
+
 
 .. js:attribute:: .general[].trap_job_errors
 

docs/manpage.rst

@@ -53,6 +53,9 @@ This is something that writers of regression tests should bear in mind.
 
    This option can also be set using the :envvar:`RFM_IGNORE_CHECK_CONFLICTS` environment variable or the :js:attr:`ignore_check_conflicts` general configuration parameter.
 
+   .. deprecated:: 3.8.0
+      This option will be removed in a future version.
+
 
 --------------
 Test filtering
@@ -790,6 +793,9 @@ Here is an alphabetical list of the environment variables recognized by ReFrame:
       Associated configuration parameter :js:attr:`ignore_check_conflicts` general configuration parameter
       ================================== ==================
 
+   .. deprecated:: 3.8.0
+      This environment variable will be removed in a future version.
+
 
 .. envvar:: RFM_TRAP_JOB_ERRORS
 

docs/tutorial_tips_tricks.rst

@@ -11,7 +11,8 @@ Debugging
 ---------
 
 ReFrame tests are Python classes inside Python source files, so the usual debugging techniques for Python apply, but the ReFrame frontend will filter some errors and stack traces by default in order to keep the output clean.
-ReFrame test files are imported, so any error that appears during import time will cause the test loading process to fail and print a stack trace pointing to the offending line.
+Generally, ReFrame will not print the full stack trace for user programming errors and will not block the test loading process.
+If a test has errors and cannot be loaded, an error message will be printed and the loading of the remaining tests will continue.
 In the following, we have inserted a small typo in the ``hello2.py`` tutorial example:
 
 .. code:: bash
@@ -20,41 +21,24 @@ In the following, we have inserted a small typo in the ``hello2.py`` tutorial ex
 
 .. code-block:: none
 
-   ./bin/reframe: name error: name 'rm' is not defined
-   ./bin/reframe: Traceback (most recent call last):
-     File "/Users/karakasv/Repositories/reframe/reframe/frontend/cli.py", line 668, in main
-       checks_found = loader.load_all()
-     File "/Users/karakasv/Repositories/reframe/reframe/frontend/loader.py", line 204, in load_all
-       checks.extend(self.load_from_dir(d, self._recurse))
-     File "/Users/karakasv/Repositories/reframe/reframe/frontend/loader.py", line 189, in load_from_dir
-       checks.extend(self.load_from_file(entry.path))
-     File "/Users/karakasv/Repositories/reframe/reframe/frontend/loader.py", line 174, in load_from_file
-       return self.load_from_module(util.import_module_from_file(filename))
-     File "/Users/karakasv/Repositories/reframe/reframe/utility/__init__.py", line 96, in import_module_from_file
-       return importlib.import_module(module_name)
-     File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
-       return _bootstrap._gcd_import(name[level:], package, level)
-     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
-     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
-     File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
-     File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
-     File "<frozen importlib._bootstrap_external>", line 728, in exec_module
-     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
-     File "/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py", line 10, in <module>
-       @rm.parameterized_test(['c'], ['cpp'])
-   NameError: name 'rm' is not defined
-
+   ./bin/reframe: skipping test file '/Users/user/Repositories/reframe/tutorials/basics/hello/hello2.py': name error: tutorials/basics/hello/hello2.py:17: name 's' is not defined
+       sanity_patterns = s.assert_found(r'Hello, World\!', 'hello.out')
+    (rerun with '-v' for more information)
+   [List of matched checks]
+   - HelloTest (found in '/Users/user/Repositories/reframe/tutorials/basics/hello/hello1.py')
+   Found 1 check(s)
 
-However, if there is a Python error inside your test's constructor, ReFrame will issue a warning and keep on loading and initializing the rest of the tests.
+Notice how ReFrame prints also the source code line that caused the error.
+This is not always the case, however.
+ReFrame cannot always track a user error back to its source and this is particularly true for the ReFrame-specific syntactic elements, such as the class `builtins <regression_test_api.html#builtins>`__.
+In such cases, ReFrame will just print the error message but not the source code context.
+In the following example, we introduce a typo in the argument of the :obj:`@run_before` decorator:
 
 .. code-block:: none
 
-   ./bin/reframe: skipping test due to errors: HelloMultiLangTest: use `-v' for more information
-     FILE: /Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py:13
-   ./bin/reframe: skipping test due to errors: HelloMultiLangTest: use `-v' for more information
-     FILE: /Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py:13
+   ./bin/reframe: skipping test file '/Users/user/Repositories/reframe/tutorials/basics/hello/hello2.py': reframe syntax error: invalid pipeline stage specified: 'compil' (rerun with '-v' for more information)
    [List of matched checks]
-   - HelloTest (found in '/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello1.py')
+   - HelloTest (found in '/Users/user/Repositories/reframe/tutorials/basics/hello/hello1.py')
    Found 1 check(s)
 
 
@@ -66,27 +50,27 @@ As suggested by the warning message, passing :option:`-v` will give you the stac
 
 .. code-block:: none
 
-   ./bin/reframe: skipping test due to errors: HelloMultiLangTest: use `-v' for more information
-     FILE: /Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py:13
-   Traceback (most recent call last):
-     File "/Users/karakasv/Repositories/reframe/reframe/core/decorators.py", line 49, in _instantiate_all
-       ret.append(_instantiate(cls, args))
-     File "/Users/karakasv/Repositories/reframe/reframe/core/decorators.py", line 32, in _instantiate
-       return cls(*args)
-     File "/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py", line 13, in __init__
-       foo
-   NameError: name 'foo' is not defined
-
-   ./bin/reframe: skipping test due to errors: HelloMultiLangTest: use `-v' for more information
-     FILE: /Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py:13
+   ./bin/reframe: skipping test file '/Users/user/Repositories/reframe/tutorials/basics/hello/hello2.py': name error: tutorials/basics/hello/hello2.py:17: name 's' is not defined
+       sanity_patterns = s.assert_found(r'Hello, World\!', 'hello.out')
+    (rerun with '-v' for more information)
    Traceback (most recent call last):
-     File "/Users/karakasv/Repositories/reframe/reframe/core/decorators.py", line 49, in _instantiate_all
-       ret.append(_instantiate(cls, args))
-     File "/Users/karakasv/Repositories/reframe/reframe/core/decorators.py", line 32, in _instantiate
-       return cls(*args)
-     File "/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello2.py", line 13, in __init__
-       foo
-   NameError: name 'foo' is not defined
+     File "/Users/user/Repositories/reframe/reframe/frontend/loader.py", line 172, in load_from_file
+       util.import_module_from_file(filename, force)
+     File "/Users/user/Repositories/reframe/reframe/utility/__init__.py", line 101, in import_module_from_file
+       return importlib.import_module(module_name)
+     File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
+       return _bootstrap._gcd_import(name[level:], package, level)
+     File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
+     File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
+     File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
+     File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
+     File "<frozen importlib._bootstrap_external>", line 790, in exec_module
+     File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
+     File "/Users/user/Repositories/reframe/tutorials/basics/hello/hello2.py", line 11, in <module>
+       class HelloMultiLangTest(rfm.RegressionTest):
+     File "/Users/user/Repositories/reframe/tutorials/basics/hello/hello2.py", line 17, in HelloMultiLangTest
+       sanity_patterns = s.assert_found(r'Hello, World\!', 'hello.out')
+   NameError: name 's' is not defined
 
    Loaded 1 test(s)
    Generated 1 test case(s)
@@ -95,9 +79,8 @@ As suggested by the warning message, passing :option:`-v` will give you the stac
    Filtering test cases(s) by other attributes: 1 remaining
    Final number of test cases: 1
    [List of matched checks]
-   - HelloTest (found in '/Users/karakasv/Repositories/reframe/tutorials/basics/hello/hello1.py')
+   - HelloTest (found in '/Users/user/Repositories/reframe/tutorials/basics/hello/hello1.py')
    Found 1 check(s)
-   Log file(s) saved in: '/var/folders/h7/k7cgrdl13r996m4dmsvjq7v80000gp/T/rfm-ckymcl44.log'
 
 
 .. tip::

reframe/core/decorators.py

@@ -21,9 +21,7 @@
 import reframe.utility.osext as osext
 import reframe.core.warnings as warn
 import reframe.core.hooks as hooks
-from reframe.core.exceptions import (ReframeSyntaxError,
-                                     SkipTestError,
-                                     user_frame)
+from reframe.core.exceptions import ReframeSyntaxError, SkipTestError, what
 from reframe.core.logging import getlogger
 from reframe.core.pipeline import RegressionTest
 from reframe.utility.versioning import VersionValidator
@@ -51,22 +49,18 @@ def _instantiate_all():
             try:
                 if cls in mod.__rfm_skip_tests:
                     continue
-
             except AttributeError:
                 mod.__rfm_skip_tests = set()
 
             try:
                 ret.append(_instantiate(cls, args))
             except SkipTestError as e:
-                getlogger().warning(f'skipping test {cls.__name__!r}: {e}')
+                getlogger().warning(f'skipping test {cls.__qualname__!r}: {e}')
             except Exception:
-                frame = user_frame(*sys.exc_info())
-                filename = frame.filename if frame else 'n/a'
-                lineno = frame.lineno if frame else 'n/a'
+                exc_info = sys.exc_info()
                 getlogger().warning(
-                    f"skipping test {cls.__name__!r} due to errors: "
-                    f"use `-v' for more information\n"
-                    f"    FILE: {filename}:{lineno}"
+                    f"skipping test {cls.__qualname__!r}: {what(*exc_info)} "
+                    f"(rerun with '-v' for more information)"
                 )
                 getlogger().verbose(traceback.format_exc())
 
@@ -88,8 +82,11 @@ def _validate_test(cls):
                                  'subclass of RegressionTest')
 
     if (cls.is_abstract()):
-        raise ValueError(f'decorated test ({cls.__qualname__!r}) has one or '
-                         f'more undefined parameters')
+        getlogger().warning(
+            f'skipping test {cls.__qualname__!r}: '
+            f'test has one or more undefined parameters'
+        )
+        return False
 
     conditions = [VersionValidator(v) for v in cls._rfm_required_version]
     if (cls._rfm_required_version and
@@ -151,7 +148,7 @@ def parameterized_test(*inst):
     def _do_register(cls):
         if _validate_test(cls):
             if not cls.param_space.is_empty():
-                raise ValueError(
+                raise ReframeSyntaxError(
                     f'{cls.__qualname__!r} is already a parameterized test'
                 )
 
@@ -204,7 +201,7 @@ def required_version(*versions):
     )
 
     if not versions:
-        raise ValueError('no versions specified')
+        raise ReframeSyntaxError('no versions specified')
 
     conditions = [VersionValidator(v) for v in versions]
 
@@ -246,10 +243,11 @@ def run_before(stage):
         from_version='3.7.0'
     )
     if stage not in _USER_PIPELINE_STAGES:
-        raise ValueError(f'invalid pipeline stage specified: {stage!r}')
+        raise ReframeSyntaxError(
+            f'invalid pipeline stage specified: {stage!r}')
 
     if stage == 'init':
-        raise ValueError('pre-init hooks are not allowed')
+        raise ReframeSyntaxError('pre-init hooks are not allowed')
 
     return hooks.attach_to('pre_' + stage)
 
@@ -268,7 +266,8 @@ def run_after(stage):
         from_version='3.7.0'
     )
     if stage not in _USER_PIPELINE_STAGES:
-        raise ValueError(f'invalid pipeline stage specified: {stage!r}')
+        raise ReframeSyntaxError(
+            f'invalid pipeline stage specified: {stage!r}')
 
     # Map user stage names to the actual pipeline functions if needed
     if stage == 'init':

reframe/core/exceptions.py

@@ -320,6 +320,22 @@ def is_exit_request(exc_type, exc_value, tb):
                                   FailureLimitError))
 
 
+def is_user_error(exc_type, exc_value, tb):
+    '''Check if error is a user programming error.
+
+    A user error is any of :py:class:`AttributeError`, :py:class:`NameError`,
+    :py:class:`TypeError` or :py:class:`ValueError` and the exception is
+    thrown from user context.
+    '''
+
+    frame = user_frame(exc_type, exc_value, tb)
+    if frame is None:
+        return False
+
+    return isinstance(exc_value,
+                      (AttributeError, NameError, TypeError, ValueError))
+
+
 def is_severe(exc_type, exc_value, tb):
     '''Check if exception is a severe one.'''
 
@@ -330,14 +346,8 @@ def is_severe(exc_type, exc_value, tb):
     if isinstance(exc_value, soft_errors):
         return False
 
-    # Treat specially type and value errors
-    type_error  = isinstance(exc_value, TypeError)
-    value_error = isinstance(exc_value, ValueError)
-    frame = user_frame(exc_type, exc_value, tb)
-    if (type_error or value_error) and frame is not None:
-        return False
-
-    return True
+    # User errors are treated as soft
+    return not is_user_error(exc_type, exc_value, tb)
 
 
 def what(exc_type, exc_value, tb):
@@ -349,14 +359,12 @@ def what(exc_type, exc_value, tb):
     reason = utility.decamelize(exc_type.__name__, ' ')
 
     # We need frame information for user type and value errors
-    frame = user_frame(exc_type, exc_value, tb)
-    user_type_error  = isinstance(exc_value, TypeError)  and frame
-    user_value_error = isinstance(exc_value, ValueError) and frame
     if isinstance(exc_value, KeyboardInterrupt):
         reason = 'cancelled by user'
     elif isinstance(exc_value, AbortTaskError):
         reason = f'aborted due to {type(exc_value.__cause__).__name__}'
-    elif user_type_error or user_value_error:
+    elif is_user_error(exc_type, exc_value, tb):
+        frame = user_frame(exc_type, exc_value, tb)
         relpath = os.path.relpath(frame.filename)
         source = ''.join(frame.code_context)
         reason += f': {relpath}:{frame.lineno}: {exc_value}\n{source}'

reframe/core/meta.py

@@ -50,7 +50,7 @@ def __setitem__(self, key, value):
                 self._namespace.pop(key, None)
 
             elif key in self['_rfm_local_param_space']:
-                raise ValueError(
+                raise ReframeSyntaxError(
                     f'cannot override parameter {key!r}'
                 )
             else:
@@ -76,7 +76,7 @@ def __getitem__(self, key):
                 except KeyError:
                     # Handle parameter access
                     if key in self['_rfm_local_param_space']:
-                        raise ValueError(
+                        raise ReframeSyntaxError(
                             'accessing a test parameter from the class '
                             'body is disallowed'
                         ) from None
@@ -426,7 +426,7 @@ class attribute. This behavior does not apply when the assigned value
                     return
                 elif not var_space[name].field is value:
                     desc = '.'.join([cls.__qualname__, name])
-                    raise ValueError(
+                    raise ReframeSyntaxError(
                         f'cannot override variable descriptor {desc!r}'
                     )
 
@@ -437,7 +437,7 @@ class attribute. This behavior does not apply when the assigned value
         try:
             param_space = super().__getattribute__('_rfm_param_space')
             if name in param_space.params:
-                raise ValueError(f'cannot override parameter {name!r}')
+                raise ReframeSyntaxError(f'cannot override parameter {name!r}')
 
         except AttributeError:
             pass

reframe/core/namespaces.py

@@ -10,6 +10,8 @@
 
 import abc
 
+from reframe.core.exceptions import ReframeSyntaxError
+
 
 class LocalNamespace:
     '''Local namespace of a regression test.
@@ -59,7 +61,7 @@ def __repr__(self):
 
     def _raise_namespace_clash(self, name):
         '''Raise an error if there is a namespace clash.'''
-        raise ValueError(
+        raise ReframeSyntaxError(
             f'{name!r} is already present in the current namespace'
         )
 
@@ -165,7 +167,7 @@ def sanity(self, cls, illegal_names=None):
 
         for key in self._namespace:
             if key in illegal_names:
-                raise ValueError(
+                raise ReframeSyntaxError(
                     f'{key!r} already defined in class '
                     f'{cls.__qualname__!r}'
                 )
@@ -177,6 +179,6 @@ def inject(self, obj, objtype=None):
         '''
 
     def __setitem__(self, key, value):
-        raise ValueError(
+        raise ReframeSyntaxError(
             f'cannot set item {key!r} into a {type(self).__qualname__} object'
         )