Job's environment is set up twice

[vkarak]

Requestor: @jgphpc
Internal issue: https://madra.cscs.ch/scs/reframe/issues/249

On dom, I get in perftools_check_c.err

perftools-base/6.4.3(43):ERROR:150: Module 'perftools-base/6.4.3' conflicts with 
the currently loaded module(s) 'perftools-lite'
perftools-base/6.4.3(43):ERROR:102: Tcl command execution failed: conflict perftools-lite

In /scratch/snx2000tds/piccinal/stage/gpu/perftools_check_c/PrgEnv-gnu/perftools_check_c_daint_gpu_PrgEnv-gnu.sh

module load daint-gpu
module unload PrgEnv-cray
module load PrgEnv-gnu
module load perftools-base
module load perftools-lite

Easy to reproduce:

module load perftools-base
module load perftools-lite
module load perftools-base

Does reframe purge the list of loaded modulefiles before submitting the slurm job ?

[vkarak]

The problem here is that the job's environment is set up twice for the submitted job:

1. Once in the setup() method and
2. once in the job script itself

So when the job is executed and tries to load perftools-base, this is already there and thus the error message.

This can be considered as a "bug" in our systems that they propagate the login environment to the compute nodes, but in systems that this is not done at all or not done properly, this is a feature. In fact, the original intention of this was this behaviour + the fact to be able to create a self-contained job submission script.

A solution to this could be to add an option at the system partition level (?) to control whether the framework will emit the corresponding environment setup options. But I'm not sure. Ideas?

[kraushm]

This issue is responsible for the error message obtained during the tests of the c2sm environment modules.
We raised the priority of the issue, since it is blocking for the c2sm-tests.

The error message encountered was:

c2sm/cray-env/base(55):ERROR:152: Module 'mvapich2_cce/2.2rc1.0.3_cuda80' is currently not       loaded
  c2sm/cray-env/base(55):ERROR:102: Tcl command execution failed: if { [module-info mode load] }  {
      module           switch mvapich2_cce/2.2rc1.0.3_cuda80 mvapich2_cce/2.2rc1.0.3
  } elseif { [module-info mode remove] } {
      module           switch mvapich2_cce/2.2rc1.0.3 mvapich2_cce/2.2rc1.0.3_cuda80
  }

[vkarak]

As discussed internally, a solution would be to take an environment snapshot inside the setup() and restore it before returning. The partition's environment and the current environment will be still loaded temporarily just for recording the exact module load/unload sequences.