[feat] Lazy evaluation of job-related attributes and environments

[vkarak]

This PR introduces two major enhancements.

Lazy evaluation of the job-related attributes

The job associated with the test is only minimally created inside setup(). Any attributes related to its run configuration (e.g., num_tasks, num_tasks_per_node etc.) are set when preparing the job script and just before launching the test. This change required that these attributes become writeable in Job's interface.

This enhancement allows you to set the num_tasks of a test based on the current partition or the environment in a cleaner way anytime after setup. For example:

@rfm.run_before('run')
def set_num_tasks(self):
    if self.current_partition.fullname == 'foo:part':
        self.num_tasks = 8
    else:
        self.num_tasks = 4

In case of flexible tests, the test's num_tasks attribute is set to the actual number of tasks assigned to it, after the test has been launched. This eliminates the need of defining a separate sanity function to retrieve the number of tasks:

# old way, but still valid

def __init__(self):
    self.sanity_patterns = sn.assert_eq(sn.count(sth), self.num_tasks_assigned)

@sn.sanity_function
@property 
def num_tasks_assigned(self):
    return self.job.num_tasks    # or now simply self.num_tasks

# new way

@rfm.run_before('sanity')
def set_sanity(self):
    self.sanity_patterns = sn.assert_eq(sn.count(sth), self.num_tasks)

Re-evaluate the test's environment whenever needed

The mechanism of loading environments has changed completely. Environment objects are now really immutable: they are just placeholders for modules and environment variables. They don't carry any state. Responsible for loading environments is the load(*environs) free function. This function returns a tuple consisting of an environment snapshot just before loading environs and the commands needed to load the environments. There is no explicit mechanism for unloading an environment as in the past. Instead, an environment or a set of environments are implicitly unloaded by restoring the environment snapshot returned by the load() function.

There is also the emit_load_commands(*environs) function, which essentially loads environs, restores the original environment and returns the required load commands. In the past this was accomplished by the _setup_environ() method of RegressionTest and the internal state of the Environment. Now, every time emit_load_commands(*environs) is called, all the environments will be loaded and the command sequence will be recorded. In addition to that, we construct a new user environment (from self.modules and self.variables) before the build and run phase of the pipeline. This allows a user to define different compilation and running environments. An example follows:

@rfm.run_before('run')
def set_num_threads(self):
    self.variables = {'OMP_NUM_THREADS': '16'}

This will define OMP_NUM_THREADS only inside the generated job script and not in the build phase.

Due to this feature, this PR fixes #58.

[victorusu]

@jenkins-cscs retest this please

[victorusu]

@jenkins-cscs retry all

[codecov-io]

Codecov Report

Merging #961 into master will increase coverage by 0.18%.
The diff coverage is 85.98%.

@@            Coverage Diff             @@
##           master     #961      +/-   ##
==========================================
+ Coverage   91.67%   91.86%   +0.18%     
==========================================
  Files          80       80              
  Lines       10575    10481      -94     
==========================================
- Hits         9695     9628      -67     
+ Misses        880      853      -27

Impacted Files	Coverage Δ
unittests/test_pipeline.py	96.71% <ø> (-0.01%)	⬇️
unittests/test_modules.py	83.17% <100%> (ø)	⬆️
reframe/core/schedulers/__init__.py	95.58% <100%> (-0.22%)	⬇️
unittests/test_schedulers.py	96.85% <100%> (ø)	⬆️
reframe/core/schedulers/pbs.py	65.88% <100%> (ø)	⬆️
unittests/test_cli.py	93.27% <100%> (ø)	⬆️
reframe/frontend/executors/policies.py	97.04% <100%> (ø)	⬆️
reframe/core/schedulers/local.py	98.87% <100%> (ø)	⬆️
reframe/frontend/cli.py	80.38% <100%> (+0.06%)	⬆️
reframe/frontend/executors/__init__.py	98.23% <100%> (ø)	⬆️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9832a6b...8d91579. Read the comment docs.