Hierarchical Module Naming Scheme with Cray toolchains

[lucamar]

I have tried following the tutorial https://easybuilders.github.io/easybuild-tutorial/hmns to implement the Hierarchical Module Naming Scheme (MNS) with a modified Cray toolchain using Lmod 8.3.1, but the compiler modulefile created under the Core folder does not update the MODULEPATH once loaded, therefore the modules created cannot be searched (not even with module spider).

I was wondering if it is at all possible to use the Hierarchical MNS with the way Cray toolchains are configured: for instance I cannot prepare the environment as suggested in the tutorial, using the commands below, or I would unload essential modules:

module purge
module unuse $MODULEPATH

Thanks for your advice!

[boegel]

@lucamar Can you share the command you run plus the contents of the Core module you obtained?

[lucamar]

Thanks for your reply @boegel. I have followed the recommended way to install EasyBuild via "pip3 install" of the tutorial, but nothing changed with respect to a local EasyBuild module. I run the command below, for instance:

eb Bison-3.7.5-cpeGNU-21.02.eb --hidden -Dr
 * [x] $CFGS/c/cpeGNU/cpeGNU-21.02.eb (module: Core | cpeGNU/21.02)
 * [x] $CFGS/m/M4/M4-1.4.18-cpeGNU-21.02.eb (module: Toolchain/cpeGNU/21.02 | M4/.1.4.18)
 * [ ] $CFGS/b/Bison/Bison-3.7.5-cpeGNU-21.02.eb (module: Toolchain/cpeGNU/21.02 | Bison/.3.7.5)

I have previously defined EASYBUILD_PREFIX=/apps/pilatus/UES/sandbox/lucamar/easybuild, therefore the folders Core and Toolchain are created under /apps/pilatus/UES/sandbox/lucamar/easybuild/modules/all. The build of cpeGNU and M4 is successful, but Bison fails since the dependency M4 cannot be loaded and the error reported by Lmod is the following:

== FAILED: Installation ended unsuccessfully (build directory: /run/user/21827/easybuild/build/Bison/3.7.5/cpeGNU-21.02): build failed (first 300 chars): Module command '/usr/share/lmod/lmod/libexec/lmod python load M4/.1.4.18' failed with exit code 1; stderr: Lmod has detected the following error: The following module(s) are unknown: "M4/.1.4.18"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "M4/.1.4.18"
Also make sure that all modulefiles written in TCL start with the string #%Module
; stdout: _mlstatus = False

The toolchain cpeGNU is a modified CrayGNU toolchain that pins the versions of Cray dependencies and loads the module cpe-gnu instead of PrgEnv-gnu, which does not exist any longer: the easyconfig file is https://github.com/lucamar/production/blob/pilatus/easybuild/easyconfigs/c/cpeGNU/cpeGNU-21.02.eb and the custom easyblock used is https://github.com/eth-cscs/production/blob/master/easybuild/easyblocks/cpetoolchain.py (cpe.py and cpegnu.py are added as local files).

If I add the command prepend_path("MODULEPATH",root:gsub("software","modules/all/Toolchain")) at the end of the Lua modulefile cpeGNU/21.02.lua, then the Toolchain path is added to MODULEPATH after the toolchain modulefile is loaded and I can load the modules as expected. Please find enclosed the PDF printout of the Lua modulefile: 21.02.lua.pdf.

You'll see that I load the Cray dependencies using modluafooter instead of dependencies: I had to do that to avoid the use of the function swap in the Lua modulefile cpeGNU/21.02.lua created by EasyBuild, since it raised a Lua error at the sanity check in some cases:

== 2021-02-18 10:41:00,170 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): Module command '/usr/share/lmod/lmod/libexec/lmod python load cpeGNU/21.02' failed with exit code 1; stderr: Lmod has detected the following error: Unable to load module because of error when evaluating modulefile:
     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua: [string "help([==[..."]:29: attempt to call a nil value (global
'swap')
     Please check the modulefile and especially if there is a the line number specified in the above message 
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    cpeGNU/21.02     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua

; stdout: _mlstatus = False
 (at easybuild/tools/modules.py:814 in run_module)

Since the Lmod installation complains often, I wonder if the issues are rather due to the Cray PE Lmod 8.3.1 : thanks for your advice!

edit: separate issue was opened for the swap problem: #3626

[boegel]

@lucamar This is using the standard HierarchicalMNS that is included with EasyBuild, correct?
If so, that's not very surprising, since the Cray toolchains are quite different from a regular toolchain...

HierarchicalMNS determines which modules should expand $MODULEPATH based on the moduleclass (both compiler and mpi are treated special there).

So in this case I think you'll need a small custom hierarchical module naming scheme specific to Cray toolchains...
That should be a lot easier to implement that HierarchicalMNS though, which has to take into account all types of special cases.

What type of module hierarchy are you interested in, just a single-level one based on toolchain version?

[lucamar]

Thanks for your feedback @boegel : yes, I have used the standard Hierarchical MNS included with EasyBuild and a hierarchy based on the toolchain name and version only.
For instance, I have created the toolchain modulefile $EASYBUILD_PREFIX/modules/all/Core/cpeGNU/21.02.lua (new toolchain cpeGNU, release 21.02) and loading it will add the folder $EASYBUILD_PREFIX/modules/all/Toolchain/cpeGNU/21.02 to the MODULEPATH, with all the modules created with the toolchain cpeGNU-21.02.
So, at present I have a working tree of hierarchical modules, using the quick and dirty workaround with the modluafooter that I pasted above prepend_path("MODULEPATH",root:gsub("software","modules/all/Toolchain")).

[boegel]

@lucamar After a bit of digging, I found the problem: the current HierarchicalMNS implementation has a hard check for toolchain names that start with Cray. That's done because we don't always have all information readily available, like the easyblock value. It's better to keep it like that, so to fix this we can simply also recognize cpe as the prefix for Cray toolchains.

In short, problem fixed in #3686 .

[lucamar]

Thanks a lot @boegel!