Enhancement: custom module trees

[verdurin]

While the CC CVMFS repos are great, I already have some cases where other modules are required. Hence it would be very nice to facilitate adding new modules with EasyBuild, and allowing the nodes to see the additional tree(s).

[mboisson]

This is already supported to some extent, but probably needs more documentation on our part. The current options are

1. eb <some recipe>.eb as a user will automatically install under $HOME/.local and this will get picked up automatically in a new shell.
2. If /opt/software/modulesfiles exists, it is automatically added to the MODULEPATH.

The first one is well integrated with our hierarchical tree, but is local to one user.
The second one is expected to be used for manually created modules.

A third option, to automatically support installing with eb into a central common location would be best to implement. This requires a bit of work on the CC stack, but should not be too hard to implement.

Any more thoughts @bartoldeman ?

[ofisette]

This is a feature I would like too. The problem with /opt/software/modulefiles is that it does not reproduce the toolchain hierarchy. If I install a package manually and make a module there for it, that works fine, but if I used eb to install a cluster-specific package, its module would end up in a toolchain directory hierarchy, and will not be made available automatically when a compiler or MPI version is loaded. This would require toolchain modules to set MODULEPATH for this specific hierarchy in addition to our CVMFS and ~/.local/easybuild/modules hierarchies. It could live in /opt/easybuild/modules.

[verdurin]

This may not be so necessary for your CC users, but for those of us elsewhere, it's not very clear which easyconfigs I can build given that you have a different scheme than the upstream default. As expected any of the upstream ones with e.g. foss toolchain don't build at least with the defaults.

For example, say I want to build VEP, with a GCC-type toolchain, is there an option?

[mboisson]

I'm not sure, but I think this specific case stems from the fact that VEP is installed with a full blown toolchain (foss) rather than GCCcore. I would think unlikely that perl modules requires MPI... I would try to bring all of those dependencies down to the GCCcore level.

[verdurin]

Okay, thanks. Will give that a try.

[mboisson]

As a potential implementation to make it possible to add MNS-aware localized module paths automatically, I wrote the following implementation a while ago :

diff --git a/lmod/SitePackage.lua b/lmod/SitePackage.lua
index c41322b..1fbf83b 100644
--- a/lmod/SitePackage.lua
+++ b/lmod/SitePackage.lua
@@ -112,11 +112,13 @@ function dofile (filename)
 	return f()
 end

-dofile("/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_logging.lua")
-dofile("/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_licenses.lua")
-dofile("/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_families.lua")
-dofile("/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_properties.lua")
-dofile("/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_visible.lua")
+local lmod_package_path = os.getenv("LMOD_PACKAGE_PATH")
+dofile(pathJoin(lmod_package_path,"SitePackage_logging.lua"))
+dofile(pathJoin(lmod_package_path,"SitePackage_licenses.lua"))
+dofile(pathJoin(lmod_package_path,"SitePackage_families.lua"))
+dofile(pathJoin(lmod_package_path,"SitePackage_properties.lua"))
+dofile(pathJoin(lmod_package_path,"SitePackage_visible.lua"))
+dofile(pathJoin(lmod_package_path,"SitePackage_localpaths.lua"))

 sandbox_registration{ loadfile = loadfile, assert = assert, loaded_modules = loaded_modules, serializeTbl = serializeTbl, clearWarningFlag = clearWarningFlag  }
 require("strict")
@@ -203,9 +205,11 @@ local function load_hook(t)
 	set_family(t)
 	default_module_change_warning(t)
 	log_module_load(t,true)
+	set_local_paths(t)
 end
 local function spider_hook(t)
 	set_props(t)
+	set_local_paths(t)
 end
 hook.register("load",           load_hook)
 hook.register("load_spider", 	spider_hook)
diff --git a/lmod/SitePackage_localpaths.lua b/lmod/SitePackage_localpaths.lua
new file mode 100644
index 0000000..3294160
--- /dev/null
+++ b/lmod/SitePackage_localpaths.lua
@@ -0,0 +1,59 @@
+require("serializeTbl")
+require("string_utils")
+
+function set_local_paths(t)
+
+	local localModulePaths = os.getenv("RSNT_LOCAL_MODULEPATHS") or nil
+	if localModulePaths == nil then
+		return
+	end
+
+
+	local moduleNamesToCheck = { "openmpi", "intelmpi", "gcc", "intel", "cuda" }
+	local myModuleName = myModuleName()
+
+	if has_value(moduleNamesToCheck, myModuleName) then
+		local arch = os.getenv("RSNT_ARCH")
+		local myModuleFullName = myModuleFullName()
+		local myFileName = myFileName()
+
+		-- get module version with two dots only
+		local myModuleVersion = myModuleVersion()
+		local myModuleVersionTwoDots = string.gsub(myModuleVersion, ".[0-9]*$", "")
+		local myModuleVersionOneDot = string.gsub(myModuleVersionTwoDots, ".[0-9]*$", "")
+
+		local rootModulePath = "^/cvmfs/soft.computecanada.ca/easybuild/modules/"
+		local relativeFileName = string.gsub(myFileName, rootModulePath, "")
+		-- from the module filename, remove the openmpi/<version>.lua
+		local relativeModulePath = string.gsub(relativeFileName, myModuleFullName .. ".lua", "")
+
+		local subPath = myModuleName .. myModuleVersionTwoDots
+		if myModuleName == "openmpi" or myModuleName == "intelmpi" or myModuleName == "impi" then
+			-- build the module path by changing Compiler or CUDA by MPI
+			relativeModulePath = string.gsub(relativeModulePath, "/Compiler/", "/MPI/")
+			relativeModulePath = string.gsub(relativeModulePath, "/CUDA/", "/MPI/")
+		elseif myModuleName == "gcc" or myModuleName == "intel" or myModuleName == "pgi" then
+			-- build the module path by changing Core by Compiler
+			relativeModulePath = string.gsub(relativeModulePath, "/Core/", "/Compiler/" .. arch)
+		elseif myModuleName == "cuda" then
+			-- build the module path by changing Compiler for CUDA
+			relativeModulePath = string.gsub(relativeModulePath, "/Compiler/", "/CUDA/")
+		end
+
+		-- intelmpi is a corner case in which we don't use the same convention
+		if myModuleName == "intelmpi" then
+			subPath = "impi" .. myModuleVersionTwoDots
+		end
+		-- and adding the subPath
+		relativeModulePath = pathJoin(relativeModulePath, subPath)
+
+		for localModulePathRoot in localModulePaths:split(":") do
+			local localModulePath = pathJoin(localModulePathRoot, relativeModulePath)
+			--LmodWarning(myModuleFullName .. ":" .. localModulePath)
+			if isDir(localModulePath) then
+				prepend_path("MODULEPATH", localModulePath)
+			end
+		end
+	end
+end
+

The idea would be that a site can specify the environment variable RSNT_LOCAL_MODULEPATHS as a secondary root for modules (i.e. it would be the same as the module-path for EasyBuild). This would inject relevant code into the compiler/mpi/cuda modules to modify the module path, and adjust the spider and load module functions accordingly.

If this is seems like a viable option, I can look at pushing this next week.

[ofisette]

RSNT_LOCAL_MODULEPATHS sounds great. Looking at the patch, I am wondering if arch should be in moduleNamesToCheck. I am not familiar with the LMod code, but loading an arch module should bring in local arch-specific modules, and I do not see how it happens with that patch. (I know I would have a mix of architectures in our local module path.)

[mboisson]

The architecture part would be taken care of due to the fact that we are replacing the root of the current module (aka /cvmfs/soft.computecanada.ca/easybuild/modules) by the local one. Everything else is relative to that, including the architecture.

[verdurin]

@mboisson yes, looks good.

The other part of this, as my earlier comments indicate, is documentation to assist with building modules for this local environment.

[mboisson]

For technical reasons, this is waiting for
#13
to be implemented (working on this today).

[mboisson]

I think I have made the required changes to support this. This is currently only on our "dev" CVMFS repository, and will not be pushed to production until further tested.

In order to test with our dev repository, you can use proot :
/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin/proot -b /cvmfs/soft-dev.computecanada.ca:/cvmfs/soft.computecanada.ca env -i CC_CLUSTER=$CC_CLUSTER TERM=$TERM HOME=$HOME bash -l

This will start a new shell in which all file access to /cvmfs/soft.computecanada.ca will be replaced by /cvmfs/soft-dev.computecanada.ca. For this, you need to also mount our soft-dev.computecanada.ca repository. You will need to modify your
/etc/cvmfs/default.local to include it :

CVMFS_REPOSITORIES="cvmfs-config.computecanada.ca,soft.computecanada.ca,soft-dev.computecanada.ca"

Once this is done, you should be able to install a recipe locally, for example :

eb --force --installpath /opt/software/easybuild SCOTCH-6.0.9-gompi-2020a.eb

and have the module it generates found if you configure this environment variable :

export RSNT_LOCAL_MODULEPATHS=/opt/software/easybuild/modules

Please test it out and let me know if it works.

If you are interested, the magic happens in

/cvmfs/soft.computecanada.ca/config/lmod/SitePackage.lua

and

/cvmfs/soft.computecanada.ca/config/lmod/SitePackage_localpaths.lua