failed load of user module with command line option --module does not cause failed tests.

[smoors]

if a module is provided on the command line, for example with --module doesnotexist, a warning is issued could not load module 'doesnotexit' correctly; skipping... but the tests continue.

reframe/reframe/frontend/cli.py

Line 1245 in 337fdcc

f'skipping...'

the safer behavior would be that the tests fail ReFrame errors out because they're not doing what the user asked, so the results are misleading.
if that's not possible (e.g. to not break user's code that depends on this), can you add an option that makes sure the tests fail?

[vkarak]

The --module option is meant to modify reframe's environment and not be part of a test. In cases, you need it as a workaround to load some modules that are needed for this system, but this should not be the case anymore. Have you tried setting the modules variable in the system configuration? Also for changing the environment of a single test before running, you can always now use the -S to set either its modules or environment variables. Let me know if that helps.

[smoors]

thanks for your reply.

Have you tried setting the modules variable in the system configuration?

I need a solution that allows to test in a non-default environment without having to modify the site config file.

Also for changing the environment of a single test before running, you can always now use the -S to set either its modules or environment variables. Let me know if that helps.

yes that works, thanks.

I still think ReFrame should error out with --module if the module cannot be loaded (without running any tests), but oh well.
I'll just avoid using the --module option then.

[ekouts]

Hi @smoors , we had an internal conversation about this and we decided that we agree with you. As it is now, it is easy to miss the warning and think you had a successful run with the module. If the module is not loaded ReFrame will fail before running any tests. We will submit a PR to fix this in the next release.