Regression test not properly removed from tasks if it fails in the cleanup stage

[teojgo]

If a regression test fails at the cleanup phase, then the cleanup phase is retried continuously.
This is a log snippet showing the cleanup phase failure:

debug: HelloWorldTestMPIOpenMP_c_static on dom:gpu using PrgEnv-cray_classic: caught builtins.FileNotFoundError: [Errno 2] No such file or directory: '/scratch/snx3000tds/jenscscs/regression/production/stage/2020-01-17_08-57-54/dom/gpu/PrgEnv-cray_classic/HelloWorldTestMPIOpenMP_c_static'
[2020-01-17T09:54:01] debug: reframe: removing task from running list: HelloWorldTestMPIOpenMP_c_static on dom:gpu using PrgEnv-cray_classic
[2020-01-17T09:54:01] debug: reframe: not in running tasks

[teojgo]

The issue can be reproduced using the following two tests:

remove_stage.py:

import reframe as rfm 
import reframe.utility.sanity as sn
import reframe.utility.os_ext as os_ext


@rfm.simple_test
class RemoveStageTest(rfm.RunOnlyRegressionTest):
    def __init__(self):
        self.valid_systems = ['dom:gpu']
        self.valid_prog_environs = ['PrgEnv-cray']
        self.sourcesdir = None
        self.executable = 'echo foo'
        self.sanity_patterns = sn.assert_found(r'foo', self.stdout)
        self.maintainers = ['TM']

    @rfm.run_after('performance')
    def rm_stage(self):
        os_ext.rmtree(self.stagedir)

and sleep_test.py:

import reframe as rfm 
import reframe.utility.sanity as sn
import reframe.utility.os_ext as os_ext


@rfm.simple_test
class SleepTest(rfm.RunOnlyRegressionTest):
    def __init__(self):
        self.valid_systems = ['dom:gpu']
        self.valid_prog_environs = ['PrgEnv-cray']
        self.sourcesdir = None
        self.executable = 'echo foo && sleep 60'
        self.sanity_patterns = sn.assert_found(r'foo', self.stdout)
        self.maintainers = ['TM']

If we use:
reframe -c sleep_test.py -c remove_stage.py --exec-policy=async -r
reframe is going to keep printing the RemoveStageTestfailure line until the SleepTest finishes

[vkarak]

@teojgo Thanks for the case. Does it fail also with the serial execution policy?

[teojgo]

@vkarak No, in the serial execution policy the test fails and the failing stage is correctly the cleanup