[feat] Add support for building tests using EasyBuild

[rsarm]

Closes #300

[pep8speaks]

Hello @rsarm, Thank you for updating!

Cheers! There are no PEP8 issues in this Pull Request!Do see the ReFrame Coding Style Guide

Comment last updated at 2021-03-08 20:43:55 UTC

[rsarm]

At this point, the test has to define the following:

        self.modules = ['EasyBuild-custom/cscs']
        self.build_system = 'EasyBuild'
        self.build_system.easyconfigs = [<easyconfigs>]
        self.build_system.options = [<easybuild-cl-options>]

[boegel]

Just some feedback, let me know if I can be of any further help with this!

[vkarak]

Thanks @boegel for reviewing!

[akesandgren]

Not sure how you've set things up here (haven't looked closely enough) but I assume it's based on having a pre-existing EB installation with the toolchains and basic stuff? Or was the plan to have it build totally from scratch?

And it needs to handle the existing EB using HMNS too.

[vkarak]

Not sure how you've set things up here (haven't looked closely enough) but I assume it's based on having a pre-existing EB installation with the toolchains and basic stuff? Or was the plan to have it build totally from scratch?

It assumes a pre-existing installation of EB.

And it needs to handle the existing EB using HMNS too.

Can you elaborate on this?

[akesandgren]

I.e., if the existing EB uses HMNS any EB config params you use must make sure to also use it, and it needs to be done in such a way that the site installed EB's modulefiles that extend MODULEPATH can find the modules you install with ReFrame.
So, first of all the site EB need to have subdir-user-modules set, so the site-installed modulefiles actually look in $HOME for HMNS modulepath extensions, then you need to have $HOME set to a dir where you can then install the ReFrame modules into that matches subdir-user-modules from the easybuild config.

Without subdir-user-modules you only get

prepend_path("MODULEPATH", "/hpc2n/eb/modules/all/Compiler/GCCcore/9.3.0")

in the site module file, with subdir-user-modules=easybuild/modules you get

prepend_path("MODULEPATH", "/hpc2n/eb/modules/all/Compiler/GCCcore/9.3.0")
if isDir(pathJoin(os.getenv("HOME"), pathJoin("easybuild/modules", "all", "Compiler/GCCcore/9.3.0"))) then
    prepend_path("MODULEPATH", pathJoin(os.getenv("HOME"), pathJoin("easybuild/modules", "all", "Compiler/GCCcore/9.3.0")))
end

without that if isDir and extra prepend_path, doing user installed EB modules with HMNS doesn't work.

So,
1 - it will only work with HMNS if the site has prepared for it
2 - you need to set $HOME and check that $HOME/$(subdir-user-modules) is usable for installing modules.

[vkarak]

Thanks @akesandgren for the explanation. The initial idea behind this feature was not to install the package, but rather stay in the stage directory just for the duration of the test. See it as testing a recipe or using EB to install and test a more complex software. If the build phase also installs it in the final location, what would happen if the sanity or the performance break? Then your test would have an undesirable side effect. Is it possible in EB to "move" an installation? Because then, users could have a post-performance hook, that would install it to its final location. @boegel @victorusu ?

[akesandgren]

That's what I mean by setting $HOME.
If the site is using HMNS for EB and has prepared for letting users use EB to install packages themselves in $HOME by setting subdir-user-modules, then if you make sure to set $HOME before running "eb some-package", it will install the modules under $HOME/ and at RunTest you can then actually load the just installed module from $HOME(set-to-some-stage-dir-of-reframe) after loading the required toolchain.

So the package ReFrame installs using EB can easily be thrown away after by just removing the stage-dir where the actual install happened.

You can't (normally) move a package installed by EB somewhere else though, due to path's usually getting upset...

Any real-life site/user installation of the package would still have to be done outside of ReFrame, and that is, as far as I'm concerned, the way it should be.

Does that explain things better?

[vkarak]

The initial idea was not to have ReFrame install the package and I'm also concerned about this, especially about the side effects I described above. We had a discussion with @rsarm and @victorusu today and we think that the best way is to build in the stage dir (unless the user really passes specific options to EB to change that, which I don't think is a good idea) and produce the necessary rpms (or any other supported packages from EB). Then, if the test is successful, you can keep this as an artifact with keep_files and deploy it by yourself either with the O/S package manager or with EB itself. This would work if you don't use RPATH installations. You could even go a step further and have a test depending on this one that would do the deployment. And, of course, you can parameterize your test on different easyconfig files ;-) There can more improvements (such as having EB spawning build jobs), but we would like to start with a minimum valuable feature and see later on what is the potential of this.

[vkarak]

We also need:

• Documentation for the EB build system
• Unit tests to test the path argument of the updated module interface
• Unit tests for the EB build system.
• We still need to update the documentation in the configuration about defining modules.

[rsarm]

Now the test should have self.modules = self.build_system.eb_modules(self.build_job.stdout). It requires to pass self.build_job.stdout which can be confusing for the users since there is self.build_stdout which is a deferrable and wouldn't work there.

[codecov-io]

Codecov Report

Merging #1719 (a3a85d2) into master (35c3020) will increase coverage by 0.01%.
The diff coverage is 80.64%.

@@            Coverage Diff             @@
##           master    #1719      +/-   ##
==========================================
+ Coverage   87.60%   87.61%   +0.01%     
==========================================
  Files          49       49              
  Lines        7940     8029      +89     
==========================================
+ Hits         6956     7035      +79     
- Misses        984      994      +10     

Impacted Files	Coverage Δ
reframe/core/runtime.py	89.15% <50.00%> (-0.48%)	⬇️
reframe/core/modules.py	56.50% <68.42%> (-0.12%)	⬇️
reframe/core/buildsystems.py	96.22% <86.48%> (-1.29%)	⬇️
reframe/core/environments.py	97.08% <100.00%> (ø)
reframe/core/pipeline.py	91.89% <100.00%> (+0.17%)	⬆️
reframe/frontend/executors/__init__.py	98.20% <0.00%> (+0.01%)	⬆️
reframe/core/schedulers/__init__.py	98.36% <0.00%> (+0.02%)	⬆️
reframe/core/logging.py	84.61% <0.00%> (+0.08%)	⬆️
reframe/utility/__init__.py	92.12% <0.00%> (+0.17%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35c3020...a3a85d2. Read the comment docs.