[feat] Add support for module collections

[vkarak]

This PR adds support for module collections for Lmod and Tmod4. Environment modules in the configuration can be defined with two alternative syntaxes: (a) simple strings as of now and (b) JSON objects with two attributes, the name of the module and whether this is a collection or not:

'modules': [{'name': 'PrgEnv-gnu', 'collection': True}]

Implementation details

• Internally, the site configuration stores the "module objects" with the extended syntax and the _SiteConfig.get() function will return that.
• A new environment property was added, modules_detailed, which returns the modules in their extended format; you can unpack them and pass them to the modules systems API.
• The modules systems backend code was improved a bit apart from the required changes and the Lmod implementation inherits from the TMod4 one, now, since both are using _mlstatus for indicating the exit status of module command.
• To better test the Lmod and Tmod backend, this PR adds two Dockerfiles with the corresponding installations. In the future we can use them in the CI.

Fixes #1526.
Replaces #1537.

[jgphpc]

👍

[pep8speaks]

Hello @vkarak, Thank you for updating!

Cheers! There are no PEP8 issues in this Pull Request!Do see the ReFrame Coding Style Guide

Comment last updated at 2020-11-09 16:55:22 UTC

[codecov-io]

Codecov Report

Merging #1586 (5ad71c7) into master (57c8d71) will decrease coverage by 0.18%.
The diff coverage is 54.87%.

@@            Coverage Diff             @@
##           master    #1586      +/-   ##
==========================================
- Coverage   87.99%   87.81%   -0.19%     
==========================================
  Files          45       45              
  Lines        7016     7106      +90     
==========================================
+ Hits         6174     6240      +66     
- Misses        842      866      +24     

Impacted Files	Coverage Δ
reframe/frontend/executors/policies.py	99.33% <ø> (-0.01%)	⬇️
reframe/core/modules.py	59.58% <41.26%> (-0.97%)	⬇️
reframe/core/config.py	91.00% <100.00%> (+0.43%)	⬆️
reframe/core/environments.py	97.05% <100.00%> (+0.35%)	⬆️
reframe/core/runtime.py	94.51% <100.00%> (ø)
reframe/frontend/cli.py	79.75% <100.00%> (+0.21%)	⬆️
reframe/utility/osext.py	92.56% <100.00%> (ø)
reframe/core/pipeline.py	92.16% <0.00%> (+0.34%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57c8d71...5ad71c7. Read the comment docs.

[vkarak]

@jenkins-cscs retry dom

[jgphpc]

test_modules.py (5ad71c7)

43 passed, 26 skipped (both tmod4 and lmod)

helloMPI: 🆗

[----------] started processing HelloWorldTestMPI_cpp_dynamic (C++ Hello World MPI Dynamic)
[ RUN      ] HelloWorldTestMPI_cpp_dynamic on eiger:mc using PrgEnv-cray
[ RUN      ] HelloWorldTestMPI_cpp_dynamic on eiger:mc using PrgEnv-gnu
[----------] finished processing HelloWorldTestMPI_cpp_dynamic (C++ Hello World MPI Dynamic)

[teojgo]

Looks fine, only the minor suggestions that I have made.

[teojgo]

Looks fine, only the minor suggestions that I have made.