PR #635 has broken Kesch environments

[vkarak]

The error is visible also in the failure of #681 here. It appears that loading the environment during setup() crashes if an environment variable contains inline commands.

[vkarak]

The setup() method essentially executes the following that triggers the error:

...
export LD_PRELOAD='$(pkg-config ...)'
module load sth

The problem is that os.expandvars for the LD_PRELOAD translates to a literal value and not to the result of the execution. This is generally harmless, since it simply prints some errors to the standard error that the specified libraries could not be found and will be ignored. The problem now is that immediately after, we execute a module command. Since we are using Tmod as backend, the only way to check for errors is to look for the ERROR string in the standard error. And there we find the error produced by the wrong LD_PRELOAD. Then ReFrame assumes that the module command has failed and produces an irrelevant error!

Actually this problem has always existe, but was never triggered. The reason is that before #635, in the setup() method we allowed users to modify the current environment. Given that the current programming environment sets LD_PRELOAD, setting self.modules would just add more modules to the environment, so that the following

self.modules = ['usermod']

would execute the following commands (variables are set after modules are loaded):

# Load the current_environ
module load progenv
module load usermod
export LD_PRELOAD=foo

#635 makes Environment immutable, so that self.modules and self.variables end up in a new environment that is loaded after the current programming environment. As a result, ReFrame executes the previous commands in different order:

# Load the current_environ
module load progenv
export LD_PRELOAD=foo

# Load the user environment
module load usermod

A solution would be to extend os.expandvars to understand the syntax $(cmd) and `cmd` and execute cmd.

[teojgo]

The problem is that os.path.expandvars does not substitute $(cmd). What would work would be something like: expanded_var = subprocess.check_output(['bash', '-c', 'echo %s' % v]).decode('utf-8').

[vkarak]

@teojgo The funny thing is how this bug was unearthed with the changes of #635.

[vkarak]

@teojgo I will provide our own version of expandvars in os_ext that will understand the syntax and run the command.