Deep copying regression tests imposes unnecessary restrictions

[vkarak]

We should rethink the fact that we deep copy the whole regression test instances for each test case (environment, system, partition). It is definitely necessary for isolating them and unties our hands in the asynchronous execution policy, but it imposes some restrictions on how we should write regression tests. Anything that cannot be deep-copied (e.g., I/O streams, generators etc.) it cannot be used in a regression test. For example, the following although it's pretty much good valid Python, we do not support it:

        self.sanity_patterns = sn.all(sn.assert_found(r'sth', fname)
                                      for fname in sn.iglob('file*.txt'))

The problem is that the argument to sn.all() is a generator and cannot be deep-copied. Instead, we should write this using a list comprehension:

        self.sanity_patterns = sn.all([sn.assert_found(r'sth', fname)
                                       for fname in sn.iglob('file*.txt')])

Internal issue: https://madra.cscs.ch/scs/reframe/issues/524

[vkarak]

The only alternative to this is to re-instantiate the check for each different programming environment and partition it runs. This has other limitations and side effects, especially, if the test modifies global variables. The only feasible solution to this problem is to improve the error messages while deep copying the test and avoid crashes.

An issue related to that is that we cannot specify the current programming environment and the current partition inside the __init__ method, because we would have to instantiate the test for all the programming environments and partitions ignoring effectively the self.valid_systems and self.valid_prog_environs. Only a drastic change to the API could solve this, by perhaps defining valid_systems and valid_prog_environs at the class level, but then supporting parameterized tests would complicate things further.

I will close this issue and I will comment in #701 about validating a regression test also in this aspect, i.e., being able to be deep copied.