Random failure of the `test_kbd_interrupt_in_wait_with_limited_concurrency` unit test

[vkarak]

The following failures happens randomly from time to time. I don't know if it actually corresponds to a bug in the code or is it a problem of this particular unit test.

_____________ test_kbd_interrupt_in_wait_with_limited_concurrency ______________
async_runner = (reframe.frontend.executors.Runner(_max_retries=0, _policy=reframe.frontend.executors.policies.AsynchronousExecutionPo...cs.TestStats(...)@0x7fdbd38f2908)@0x7fdbd38f2c18, <unittests.test_policies._TaskEventMonitor object at 0x7fdbd38f2208>)
make_cases = <function make_cases.<locals>._make_cases at 0x7fdbd34edc80>
make_async_exec_ctx = <function make_async_exec_ctx.<locals>._make_async_exec_ctx at 0x7fdbd34f46a8>
    def test_kbd_interrupt_in_wait_with_limited_concurrency(
            async_runner, make_cases, make_async_exec_ctx):
        # The general idea for this test is to allow enough time for all the
        # four checks to be submitted and at the same time we need the
        # KeyboardInterruptCheck to finish first (the corresponding wait should
        # trigger the failure), so as to make the framework kill the remaining
        # three.
        ctx = make_async_exec_ctx(2)
        next(ctx)
    
        runner, _ = async_runner
        with pytest.raises(KeyboardInterrupt):
            runner.runall(make_cases([
                KeyboardInterruptCheck(), SleepCheck(10),
>               SleepCheck(10), SleepCheck(10)
            ]))
E           Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
unittests/test_policies.py:525: Failed
=========================== short test summary info ============================
FAILED unittests/test_policies.py::test_kbd_interrupt_in_wait_with_limited_concurrency
================== 1 failed, 619 passed, 88 skipped in 38.85s ==================

[vkarak]

It seems that this failure appears only with Python 3.7 and 3.8.

[vkarak]

This is a very strange error. It now appears with Python 3.6 and not with 3.7 or 3.8. Also, I couldn't find any scenario that would cause KeyboardInterrupt not to be thrown. That would mean that somehow the task associated with the KeyboardInterruptCheck is not waited for, which I couldn't find any execution path that would make this happen.

[vkarak]

Full dump from failure:

Runner({
    '_policy': AsynchronousExecutionPolicy({
        'skip_system_check': False,
        'force_local': False,
        'skip_environ_check': False,
        'skip_sanity_check': False,
        'skip_performance_check': False,
        'keep_stage_files': True,
        'only_environs': None,
        'printer': PrettyPrinter({
            'colorize': True,
            'line_width': 78,
            'status_width': 10,
            '_progress_count': 4,
            '_progress_total': 4
        })@0x7f524c8a8c18,
        'strict_check': False,
        'sched_flex_alloc_nodes': None,
        'sched_account': None,
        'sched_partition': None,
        'sched_reservation': None,
        'sched_nodelist': None,
        'sched_exclude_nodelist': None,
        'sched_options': [],
        'task_listeners': [
            AsynchronousExecutionPolicy(...)@0x7f524c8a85c0,
            _TaskEventMonitor({
                'num_tasks': [
                    0
                ],
                'tasks': []
            })@0x7f524c8a8898
        ],
        'stats': TestStats({
            '_tasks': [
                [
                    RegressionTask({
                        '_case': TestCase({
                            '_TestCase__check_orig': KeyboardInterruptCheck({
                                'name': 'KeyboardInterruptCheck',
                                'descr': 'KeyboardInterruptCheck',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'sleep 1',
                                'executable_opts': [],
                                'prerun_cmds': [],
                                'postrun_cmds': [],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'KeyboardInterruptCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': 'src',
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function({})@0x7f524d194620,
                                    '_args': (
                                        'hello',
                                        _DeferredExpression({
                                            '_fn': function({})@0x7f524d1d28c8,
                                            '_args': (
                                                KeyboardInterruptCheck(...)@0x7f524c5e6080
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524cb83c18
                                    ),
                                    '_kwargs': {}
                                })@0x7f524cb83cc0,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c5e64a8,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': None,
                                '_current_environ': None,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter({
                                    'logger': None,
                                    'extra': {
                                        'check_name': 'reframe',
                                        'check_jobid': '-1',
                                        'check_job_completion_time': None,
                                        'check_job_completion_time_unix': None,
                                        'check_info': 'reframe',
                                        'check_system': None,
                                        'check_partition': None,
                                        'check_environ': None,
                                        'check_outputdir': None,
                                        'check_stagedir': None,
                                        'check_num_tasks': None,
                                        'check_perf_var': None,
                                        'check_perf_value': None,
                                        'check_perf_ref': None,
                                        'check_perf_lower_thres': None,
                                        'check_perf_upper_thres': None,
                                        'check_perf_unit': None,
                                        'osuser': 'travis',
                                        'osgroup': 'travis',
                                        'check_tags': None,
                                        'version': '3.1-dev2 (rev: b2d3b441)'
                                    },
                                    'check': None,
                                    'colorize': False
                                })@0x7f524e76aac8,
                                '_userdeps': [],
                                '_case': None,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524cb837f0,
                                'phase': 'wait'
                            })@0x7f524c5e6080,
                            '_TestCase__check': KeyboardInterruptCheck({
                                'name': 'KeyboardInterruptCheck',
                                'descr': 'KeyboardInterruptCheck',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'sleep 1',
                                'executable_opts': [],
                                'prerun_cmds': [],
                                'postrun_cmds': [],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'KeyboardInterruptCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': 'src',
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        'hello',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                KeyboardInterruptCheck(...)@0x7f524ca05748
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c9cec50
                                    ),
                                    '_kwargs': {}
                                })@0x7f524ca054e0,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c9ceba8,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': SystemPartition({
                                    '_parent_system': 'generic',
                                    '_name': 'default',
                                    '_scheduler': ABCMeta(mappingproxy({'__module__': 'reframe.core.schedulers.local', '__init__': <function LocalJobScheduler.__init__ at 0x7f524d1ee378>, 'completion_time': <function LocalJobScheduler.completion_time at 0x7f524d1ee400>, 'submit': <function LocalJobScheduler.submit at 0x7f524d1ee488>, 'emit_preamble': <function LocalJobScheduler.emit_preamble at 0x7f524d1ee510>, 'allnodes': <function LocalJobScheduler.allnodes at 0x7f524d1ee598>, 'filternodes': <function LocalJobScheduler.filternodes at 0x7f524d1ee620>, '_kill_all': <function LocalJobScheduler._kill_all at 0x7f524d1ee6a8>, '_term_all': <function LocalJobScheduler._term_all at 0x7f524d1ee730>, '_wait_all': <function LocalJobScheduler._wait_all at 0x7f524d1ee7b8>, 'cancel': <function LocalJobScheduler.cancel at 0x7f524d1ee840>, 'wait': <function LocalJobScheduler.wait at 0x7f524d1ee8c8>, 'finished': <function LocalJobScheduler.finished at 0x7f524d1ee950>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object at 0x7f524d1f03c0>, 'is_local': <reframe.core.fields.ConstantField object at 0x7f524e7c7438>, 'registered_name': <reframe.core.fields.ConstantField object at 0x7f524d205dd8>}))@0x196ed18,
                                    '_launcher': ABCMeta(mappingproxy({'__module__': 'reframe.core.launchers.local', 'command': <function LocalLauncher.command at 0x7f524d15c8c8>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object at 0x7f524d1f0660>, 'is_local': <reframe.core.fields.ConstantField object at 0x7f524d15b9e8>, 'registered_name': <reframe.core.fields.ConstantField object at 0x7f524d1687f0>}))@0x198e528,
                                    '_descr': 'Login nodes',
                                    '_access': [],
                                    '_container_environs': {},
                                    '_local_env': Environment({
                                        '_name': '__rfm_env_default',
                                        '_modules': [],
                                        '_variables': {}
                                    })@0x7f524cf393c8,
                                    '_environs': [
                                        ProgEnvironment({
                                            '_name': 'builtin-gcc',
                                            '_modules': [],
                                            '_variables': {},
                                            '_cc': 'gcc',
                                            '_cxx': 'g++',
                                            '_ftn': 'gfortran',
                                            '_nvcc': 'nvcc',
                                            '_cppflags': [],
                                            '_cflags': [],
                                            '_cxxflags': [],
                                            '_fflags': [],
                                            '_ldflags': []
                                        })@0x7f524cae56d8
                                    ],
                                    '_max_jobs': 2,
                                    '_resources': {}
                                })@0x7f524ca05668,
                                '_current_environ': ProgEnvironment({
                                    '_name': 'builtin-gcc',
                                    '_modules': [],
                                    '_variables': {},
                                    '_cc': 'gcc',
                                    '_cxx': 'g++',
                                    '_ftn': 'gfortran',
                                    '_nvcc': 'nvcc',
                                    '_cppflags': [],
                                    '_cflags': [],
                                    '_cxxflags': [],
                                    '_fflags': [],
                                    '_ldflags': []
                                })@0x7f524c588f98,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter({
                                    'logger': None,
                                    'extra': {
                                        'check_name': 'reframe',
                                        'check_jobid': '-1',
                                        'check_job_completion_time': None,
                                        'check_job_completion_time_unix': None,
                                        'check_info': 'reframe',
                                        'check_system': None,
                                        'check_partition': None,
                                        'check_environ': None,
                                        'check_outputdir': None,
                                        'check_stagedir': None,
                                        'check_num_tasks': None,
                                        'check_perf_var': None,
                                        'check_perf_value': None,
                                        'check_perf_ref': None,
                                        'check_perf_lower_thres': None,
                                        'check_perf_upper_thres': None,
                                        'check_perf_unit': None,
                                        'osuser': 'travis',
                                        'osgroup': 'travis',
                                        'check_tags': None,
                                        'version': '3.1-dev2 (rev: b2d3b441)'
                                    },
                                    'check': None,
                                    'colorize': False
                                })@0x7f524c9da278,
                                '_userdeps': [],
                                '_case': <weakref at 0x7f524c5cd598; to 'TestCase' at 0x7f524ca05c88>,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524cf39588,
                                'phase': 'wait'
                            })@0x7f524ca05748,
                            '_TestCase__partition': SystemPartition(...)@0x7f524ca05668,
                            '_TestCase__environ': ProgEnvironment(...)@0x7f524c588f98,
                            '_TestCase__deps': [],
                            'in_degree': 0
                        })@0x7f524ca05c88,
                        '_failed_stage': 'setup',
                        '_current_stage': 'setup',
                        '_exc_info': (
                            type(mappingproxy({'__module__': 'reframe.core.exceptions', '__doc__': 'Raised when a condition prevents the regression test pipeline to\n    continue and the error may not be described by another more specific\n    exception.\n    '}))@0x18a2b28,
                            PipelineError({
                                '_message': 'failed to set up paths'
                            })@0x7f5255cc8048,
                            <traceback object at 0x7f524c916a88>
                        ),
                        '_listeners': [
                            AsynchronousExecutionPolicy(...)@0x7f524c8a85c0,
                            _TaskEventMonitor(...)@0x7f524c8a8898
                        ],
                        'ref_count': 0,
                        'zombie': False,
                        '_timestamps': {
                            'setup_start': 1594477242.3653297,
                            'setup_finish': 1594477242.3658,
                            'pipeline_end': 1594477242.3658016
                        }
                    })@0x7f524c5e60b8,
                    RegressionTask({
                        '_case': TestCase({
                            '_TestCase__check_orig': SleepCheck({
                                'name': 'SleepCheck_10_14',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524cb838d0
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c9cd780
                                    ),
                                    '_kwargs': {}
                                })@0x7f524c9cd5c0,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524cb839b0,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': None,
                                '_current_environ': None,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter(...)@0x7f524e76aac8,
                                '_userdeps': [],
                                '_case': None,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524c9cd080,
                                'sleep_time': 10
                            })@0x7f524cb838d0,
                            '_TestCase__check': SleepCheck({
                                'name': 'SleepCheck_10_14',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524c588320
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c588e80
                                    ),
                                    '_kwargs': {}
                                })@0x7f524c588198,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c588208,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': SystemPartition({
                                    '_parent_system': 'generic',
                                    '_name': 'default',
                                    '_scheduler': ABCMeta(...)@0x196ed18,
                                    '_launcher': ABCMeta(...)@0x198e528,
                                    '_descr': 'Login nodes',
                                    '_access': [],
                                    '_container_environs': {},
                                    '_local_env': Environment({
                                        '_name': '__rfm_env_default',
                                        '_modules': [],
                                        '_variables': {}
                                    })@0x7f524c588748,
                                    '_environs': [
                                        ProgEnvironment({
                                            '_name': 'builtin-gcc',
                                            '_modules': [],
                                            '_variables': {},
                                            '_cc': 'gcc',
                                            '_cxx': 'g++',
                                            '_ftn': 'gfortran',
                                            '_nvcc': 'nvcc',
                                            '_cppflags': [],
                                            '_cflags': [],
                                            '_cxxflags': [],
                                            '_fflags': [],
                                            '_ldflags': []
                                        })@0x7f524c588908
                                    ],
                                    '_max_jobs': 2,
                                    '_resources': {}
                                })@0x7f524c588128,
                                '_current_environ': ProgEnvironment({
                                    '_name': 'builtin-gcc',
                                    '_modules': [],
                                    '_variables': {},
                                    '_cc': 'gcc',
                                    '_cxx': 'g++',
                                    '_ftn': 'gfortran',
                                    '_nvcc': 'nvcc',
                                    '_cppflags': [],
                                    '_cflags': [],
                                    '_cxxflags': [],
                                    '_fflags': [],
                                    '_ldflags': []
                                })@0x7f524c5886d8,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter({
                                    'logger': None,
                                    'extra': {
                                        'check_name': 'reframe',
                                        'check_jobid': '-1',
                                        'check_job_completion_time': None,
                                        'check_job_completion_time_unix': None,
                                        'check_info': 'reframe',
                                        'check_system': None,
                                        'check_partition': None,
                                        'check_environ': None,
                                        'check_outputdir': None,
                                        'check_stagedir': None,
                                        'check_num_tasks': None,
                                        'check_perf_var': None,
                                        'check_perf_value': None,
                                        'check_perf_ref': None,
                                        'check_perf_lower_thres': None,
                                        'check_perf_upper_thres': None,
                                        'check_perf_unit': None,
                                        'osuser': 'travis',
                                        'osgroup': 'travis',
                                        'check_tags': None,
                                        'version': '3.1-dev2 (rev: b2d3b441)'
                                    },
                                    'check': None,
                                    'colorize': False
                                })@0x7f524c588828,
                                '_userdeps': [],
                                '_case': <weakref at 0x7f524c73c228; to 'TestCase' at 0x7f524ca053c8>,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524c588cf8,
                                'sleep_time': 10
                            })@0x7f524c588320,
                            '_TestCase__partition': SystemPartition(...)@0x7f524c588128,
                            '_TestCase__environ': ProgEnvironment(...)@0x7f524c5886d8,
                            '_TestCase__deps': [],
                            'in_degree': 0
                        })@0x7f524ca053c8,
                        '_failed_stage': 'setup',
                        '_current_stage': 'setup',
                        '_exc_info': (
                            type(...)@0x18a2b28,
                            PipelineError({
                                '_message': 'failed to set up paths'
                            })@0x7f524c6a9f48,
                            <traceback object at 0x7f524c916bc8>
                        ),
                        '_listeners': [
                            AsynchronousExecutionPolicy(...)@0x7f524c8a85c0,
                            _TaskEventMonitor(...)@0x7f524c8a8898
                        ],
                        'ref_count': 0,
                        'zombie': False,
                        '_timestamps': {
                            'setup_start': 1594477242.369037,
                            'setup_finish': 1594477242.3694599,
                            'pipeline_end': 1594477242.3694618
                        }
                    })@0x7f524c5e69e8,
                    RegressionTask({
                        '_case': TestCase({
                            '_TestCase__check_orig': SleepCheck({
                                'name': 'SleepCheck_10_15',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524cb832b0
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c9cda58
                                    ),
                                    '_kwargs': {}
                                })@0x7f524c9cd748,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c9cd8d0,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': None,
                                '_current_environ': None,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter(...)@0x7f524e76aac8,
                                '_userdeps': [],
                                '_case': None,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524c9cd710,
                                'sleep_time': 10
                            })@0x7f524cb832b0,
                            '_TestCase__check': SleepCheck({
                                'name': 'SleepCheck_10_15',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524c5882b0
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c602b38
                                    ),
                                    '_kwargs': {}
                                })@0x7f524c6022e8,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c602e48,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': SystemPartition({
                                    '_parent_system': 'generic',
                                    '_name': 'default',
                                    '_scheduler': ABCMeta(...)@0x196ed18,
                                    '_launcher': ABCMeta(...)@0x198e528,
                                    '_descr': 'Login nodes',
                                    '_access': [],
                                    '_container_environs': {},
                                    '_local_env': Environment({
                                        '_name': '__rfm_env_default',
                                        '_modules': [],
                                        '_variables': {}
                                    })@0x7f524cac86d8,
                                    '_environs': [
                                        ProgEnvironment({
                                            '_name': 'builtin-gcc',
                                            '_modules': [],
                                            '_variables': {},
                                            '_cc': 'gcc',
                                            '_cxx': 'g++',
                                            '_ftn': 'gfortran',
                                            '_nvcc': 'nvcc',
                                            '_cppflags': [],
                                            '_cflags': [],
                                            '_cxxflags': [],
                                            '_fflags': [],
                                            '_ldflags': []
                                        })@0x7f524cb8ef28
                                    ],
                                    '_max_jobs': 2,
                                    '_resources': {}
                                })@0x7f524c6c6c18,
                                '_current_environ': ProgEnvironment({
                                    '_name': 'builtin-gcc',
                                    '_modules': [],
                                    '_variables': {},
                                    '_cc': 'gcc',
                                    '_cxx': 'g++',
                                    '_ftn': 'gfortran',
                                    '_nvcc': 'nvcc',
                                    '_cppflags': [],
                                    '_cflags': [],
                                    '_cxxflags': [],
                                    '_fflags': [],
                                    '_ldflags': []
                                })@0x7f524cb8ecf8,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter({
                                    'logger': None,
                                    'extra': {
                                        'check_name': 'reframe',
                                        'check_jobid': '-1',
                                        'check_job_completion_time': None,
                                        'check_job_completion_time_unix': None,
                                        'check_info': 'reframe',
                                        'check_system': None,
                                        'check_partition': None,
                                        'check_environ': None,
                                        'check_outputdir': None,
                                        'check_stagedir': None,
                                        'check_num_tasks': None,
                                        'check_perf_var': None,
                                        'check_perf_value': None,
                                        'check_perf_ref': None,
                                        'check_perf_lower_thres': None,
                                        'check_perf_upper_thres': None,
                                        'check_perf_unit': None,
                                        'osuser': 'travis',
                                        'osgroup': 'travis',
                                        'check_tags': None,
                                        'version': '3.1-dev2 (rev: b2d3b441)'
                                    },
                                    'check': None,
                                    'colorize': False
                                })@0x7f524cac8ef0,
                                '_userdeps': [],
                                '_case': <weakref at 0x7f524d1cacc8; to 'TestCase' at 0x7f524c588470>,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524cac8f60,
                                'sleep_time': 10
                            })@0x7f524c5882b0,
                            '_TestCase__partition': SystemPartition(...)@0x7f524c6c6c18,
                            '_TestCase__environ': ProgEnvironment(...)@0x7f524cb8ecf8,
                            '_TestCase__deps': [],
                            'in_degree': 0
                        })@0x7f524c588470,
                        '_failed_stage': 'setup',
                        '_current_stage': 'setup',
                        '_exc_info': (
                            type(...)@0x18a2b28,
                            PipelineError({
                                '_message': 'failed to set up paths'
                            })@0x7f524c6a9e28,
                            <traceback object at 0x7f524c748f88>
                        ),
                        '_listeners': [
                            AsynchronousExecutionPolicy(...)@0x7f524c8a85c0,
                            _TaskEventMonitor(...)@0x7f524c8a8898
                        ],
                        'ref_count': 0,
                        'zombie': False,
                        '_timestamps': {
                            'setup_start': 1594477242.3727252,
                            'setup_finish': 1594477242.3731358,
                            'pipeline_end': 1594477242.3731375
                        }
                    })@0x7f524c5e6828,
                    RegressionTask({
                        '_case': TestCase({
                            '_TestCase__check_orig': SleepCheck({
                                'name': 'SleepCheck_10_16',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524c9cda90
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524ca05a20
                                    ),
                                    '_kwargs': {}
                                })@0x7f524ca05710,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c9cd668,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': None,
                                '_current_environ': None,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter(...)@0x7f524e76aac8,
                                '_userdeps': [],
                                '_case': None,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524ca05048,
                                'sleep_time': 10
                            })@0x7f524c9cda90,
                            '_TestCase__check': SleepCheck({
                                'name': 'SleepCheck_10_16',
                                'descr': 'SleepCheck_10',
                                'valid_prog_environs': [
                                    '*'
                                ],
                                'valid_systems': [
                                    '*'
                                ],
                                'sourcepath': '',
                                'prebuild_cmds': [],
                                'postbuild_cmds': [],
                                'executable': 'python3',
                                'executable_opts': [
                                    '-c "from time import sleep; sleep(10)"'
                                ],
                                'prerun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'postrun_cmds': [
                                    'python3 -c "from datetime import datetime; print(datetime.today().strftime(\'%s.%f\'), flush=True)"'
                                ],
                                'keep_files': [],
                                'readonly_files': [],
                                'tags': {
                                    'SleepCheck'
                                },
                                'maintainers': [
                                    'VK'
                                ],
                                '_perfvalues': {},
                                'container_platform': None,
                                'strict_check': True,
                                'num_tasks': 1,
                                'num_tasks_per_node': None,
                                'num_gpus_per_node': 0,
                                'num_cpus_per_task': None,
                                'num_tasks_per_core': None,
                                'num_tasks_per_socket': None,
                                'use_multithreading': None,
                                'exclusive_access': False,
                                'max_pending_time': None,
                                'local': True,
                                '_prefix': '/home/travis/build/eth-cscs/reframe/unittests/resources/checks',
                                'sourcesdir': None,
                                'sanity_patterns': _DeferredExpression({
                                    '_fn': function(...)@0x7f524d194620,
                                    '_args': (
                                        '.*',
                                        _DeferredExpression({
                                            '_fn': function(...)@0x7f524d1d28c8,
                                            '_args': (
                                                SleepCheck(...)@0x7f524cb8fe10
                                            ),
                                            '_kwargs': {}
                                        })@0x7f524c58fe48
                                    ),
                                    '_kwargs': {}
                                })@0x7f524c58f5f8,
                                'perf_patterns': None,
                                'reference': ScopedDict({
                                    'data': {},
                                    '_scope_sep': ':',
                                    '_global_scope': '*'
                                })@0x7f524c58f828,
                                'modules': [],
                                'variables': {},
                                'time_limit': datetime.timedelta(seconds=600),
                                '_current_partition': SystemPartition({
                                    '_parent_system': 'generic',
                                    '_name': 'default',
                                    '_scheduler': ABCMeta(...)@0x196ed18,
                                    '_launcher': ABCMeta(...)@0x198e528,
                                    '_descr': 'Login nodes',
                                    '_access': [],
                                    '_container_environs': {},
                                    '_local_env': Environment({
                                        '_name': '__rfm_env_default',
                                        '_modules': [],
                                        '_variables': {}
                                    })@0x7f524ca61cc0,
                                    '_environs': [
                                        ProgEnvironment({
                                            '_name': 'builtin-gcc',
                                            '_modules': [],
                                            '_variables': {},
                                            '_cc': 'gcc',
                                            '_cxx': 'g++',
                                            '_ftn': 'gfortran',
                                            '_nvcc': 'nvcc',
                                            '_cppflags': [],
                                            '_cflags': [],
                                            '_cxxflags': [],
                                            '_fflags': [],
                                            '_ldflags': []
                                        })@0x7f524ca58be0
                                    ],
                                    '_max_jobs': 2,
                                    '_resources': {}
                                })@0x7f524c58feb8,
                                '_current_environ': ProgEnvironment({
                                    '_name': 'builtin-gcc',
                                    '_modules': [],
                                    '_variables': {},
                                    '_cc': 'gcc',
                                    '_cxx': 'g++',
                                    '_ftn': 'gfortran',
                                    '_nvcc': 'nvcc',
                                    '_cppflags': [],
                                    '_cflags': [],
                                    '_cxxflags': [],
                                    '_fflags': [],
                                    '_ldflags': []
                                })@0x7f524ca58cc0,
                                '_job': None,
                                'extra_resources': {},
                                '_stagedir': None,
                                '_outputdir': None,
                                '_stdout': None,
                                '_stderr': None,
                                '_build_job': None,
                                '_compile_proc': None,
                                'build_system': None,
                                '_perf_logger': LoggerAdapter({
                                    'logger': None,
                                    'extra': {
                                        'check_name': 'reframe',
                                        'check_jobid': '-1',
                                        'check_job_completion_time': None,
                                        'check_job_completion_time_unix': None,
                                        'check_info': 'reframe',
                                        'check_system': None,
                                        'check_partition': None,
                                        'check_environ': None,
                                        'check_outputdir': None,
                                        'check_stagedir': None,
                                        'check_num_tasks': None,
                                        'check_perf_var': None,
                                        'check_perf_value': None,
                                        'check_perf_ref': None,
                                        'check_perf_lower_thres': None,
                                        'check_perf_upper_thres': None,
                                        'check_perf_unit': None,
                                        'osuser': 'travis',
                                        'osgroup': 'travis',
                                        'check_tags': None,
                                        'version': '3.1-dev2 (rev: b2d3b441)'
                                    },
                                    'check': None,
                                    'colorize': False
                                })@0x7f524ca61860,
                                '_userdeps': [],
                                '_case': <weakref at 0x7f524c8bebd8; to 'TestCase' at 0x7f524cb8ea58>,
                                '_cdt_environ': Environment({
                                    '_name': '__rfm_cdt_environ',
                                    '_modules': [],
                                    '_variables': {}
                                })@0x7f524ca61550,
                                'sleep_time': 10
                            })@0x7f524cb8fe10,
                            '_TestCase__partition': SystemPartition(...)@0x7f524c58feb8,
                            '_TestCase__environ': ProgEnvironment(...)@0x7f524ca58cc0,
                            '_TestCase__deps': [],
                            'in_degree': 0
                        })@0x7f524cb8ea58,
                        '_failed_stage': 'setup',
                        '_current_stage': 'setup',
                        '_exc_info': (
                            type(...)@0x18a2b28,
                            PipelineError({
                                '_message': 'failed to set up paths'
                            })@0x7f524c6a9d68,
                            <traceback object at 0x7f524cbb1688>
                        ),
                        '_listeners': [
                            AsynchronousExecutionPolicy(...)@0x7f524c8a85c0,
                            _TaskEventMonitor(...)@0x7f524c8a8898
                        ],
                        'ref_count': 0,
                        'zombie': False,
                        '_timestamps': {
                            'setup_start': 1594477242.376323,
                            'setup_finish': 1594477242.3767543,
                            'pipeline_end': 1594477242.376756
                        }
                    })@0x7f524c9e8940
                ]
            ]
        })@0x7f524c8a8630,
        '_task_index': {
            TestCase(...)@0x7f524ca05c88: RegressionTask(...)@0x7f524c5e60b8,
            TestCase(...)@0x7f524ca053c8: RegressionTask(...)@0x7f524c5e69e8,
            TestCase(...)@0x7f524c588470: RegressionTask(...)@0x7f524c5e6828,
            TestCase(...)@0x7f524cb8ea58: RegressionTask(...)@0x7f524c9e8940
        },
        '_running_tasks': [],
        '_completed_tasks': [],
        '_retired_tasks': [],
        '_running_tasks_counts': {
            'generic:default': 0
        },
        '_ready_tasks': {
            'generic:default': []
        },
        '_waiting_tasks': [],
        '_max_jobs': {
            'generic:default': 2
        }
    })@0x7f524c8a85c0,
    '_printer': PrettyPrinter(...)@0x7f524c8a8c18,
    '_max_retries': 0,
    '_stats': TestStats(...)@0x7f524c8a8630
})@0x7f524c8a8ba8

[vkarak]

Interesting! KeyboardInterruptCheck fails in setup()before it raises KeyboardInterrupt:

                        '_exc_info': (
                            type(mappingproxy({'__module__': 'reframe.core.exceptions', '__doc__': 'Raised when a condition prevents the regression test pipeline to\n    continue and the error may not be described by another more specific\n    exception.\n    '}))@0x18a2b28,
                            PipelineError({
                                '_message': 'failed to set up paths'
                            })@0x7f5255cc8048,
                            <traceback object at 0x7f524c916a88>
                        ),

[vkarak]

It seems that it fails to create the stage or output directory of this test.