ReFrame throws `SyntaxError` when loading some some Cray `cdt` modules

[bcfriesen]

Cray ships cdt modules which sets the default version for a bundle of Cray user software which includes the CCE compiler, CrayPat, etc.

Under 'normal' circumstances, loading a particular cdt module has the following effect:

user@cori03:~> module load cdt/19.06
Switching to atp/2.1.3.
Switching to cray-libsci/19.06.1.
Switching to cray-mpich/7.7.8.
Switching to craype/2.6.0.
Switching to modules/3.2.11.2.
Switching to pmi/5.0.14.

I would like to run a large ReFrame test suite using different versions of the cdt module. However, when I attempt to execute:

reframe <many options> -m cdt/19.03

It throws this error:

Reframe version: 2.19-nersc
Launched by user: friesen
Launched on host: cori03
Reframe paths
=============
    Check prefix      :
(R) Check search path : './basic_functionality/time_test'
    Stage dir prefix     : /global/cscratch1/sd/friesen/system_regtest/cori/stage/
    Output dir prefix    : /global/homes/f/friesen/system_regtest/cori/output/
    Perf. logging prefix : /global/homes/f/friesen/system_regtest/cori/log
/global/homes/f/friesen/reframe/reframe.git/bin/reframe: unexpected error: invalid syntax (<string>, line 1)
Traceback (most recent call last):
  File "/global/u2/f/friesen/reframe/reframe.git/reframe/frontend/cli.py", line 508, in main
    rt.modules_system.load_module(m, force=True)
  File "/global/u2/f/friesen/reframe/reframe.git/reframe/core/modules.py", line 184, in load_module
    ret += self._load_module(m, force)
  File "/global/u2/f/friesen/reframe/reframe.git/reframe/core/modules.py", line 204, in _load_module
    self._backend.load_module(module)
  File "/global/u2/f/friesen/reframe/reframe.git/reframe/core/modules.py", line 476, in load_module
    msg="could not load module '%s' correctly" % module)
  File "/global/u2/f/friesen/reframe/reframe.git/reframe/core/modules.py", line 453, in _exec_module_command
    exec(completed.stdout)
  File "<string>", line 1
    source /opt/cray/pe/modules/3.2.11.1/init/bash
                                     ^
SyntaxError: invalid syntax

Curiously, this error does not occur if I use the cdt/19.06 module.

Any ideas what's going on?

I noticed this looks vaguely similar to #895, but not exactly the same.

[vkarak]

Hi @bcfriesen, we came across this and it's a Cray bug (check case #243415). The Python wrappers in the recent modules from CLE7 do not emit valid Python code. They actually emit this source /opt/cray/pe/modules/3.2.11.1/init/bash line (which apparently comes from the bash wrapper) plus the actual Python commands. This came as as a fix to an older bug, that switching the modules module didn't actually work. According to Cray this behaviour is fixed in PE19.11. For the moment, the workaround is to load cdt before running ReFrame and without using the -m option.

Also in order to test correctly a non-default PE, you should pass the --non-default-craype option, which was introduced in #949 (from 2.20-dev1 onward).

[bcfriesen]

Makes sense, thanks @vkarak. Since this is a known issue with a plan to fix in the future from Cray, you can close this.