add support for --update-lmod-caches (disabled by default) and --compile-lmod-caches (WIP) #1177
Conversation
…ame, build all cache types (moduleT, dbT, reverseMapT)
|
Refer to this link for build results (access rights to CI server needed): |
easybuild/tools/modules.py
Outdated
| def compile_cache(self, cache_fp): | ||
| """Compile Lmod cache file using luac.""" | ||
| if build_option('compile_lmod_caches'): | ||
| # FIXME: ask Lmod to compile, to ensure that luac that matches the lua used by Lmod is used? |
There was a problem hiding this comment.
👍 on letting lmod compile the cache. then we also don't need to test if luac is in PATH etc etc.
There was a problem hiding this comment.
that's just a side-effect, but yes, that's good
it's important that Lmod uses the luac that matches the lua it's using, since the Lua version has to be part of the filename of the compiled cache (I still need to fix that here)
|
Can we look into drop using $LMOD_IGNORE_CACHE=1 globally? |
easybuild/tools/modules.py
Outdated
| from easybuild.tools.module_naming_scheme import DEVEL_MODULE_SUFFIX | ||
| from easybuild.tools.run import run_cmd | ||
| from easybuild.tools.toolchain import DUMMY_TOOLCHAIN_NAME, DUMMY_TOOLCHAIN_VERSION | ||
| from vsc.utils.missing import nub | ||
|
|
||
| # relative to user's home directory | ||
| LMOD_USER_CACHE_RELDIR = '.lmod.d/.cache' |
There was a problem hiding this comment.
--with-useDotFiles=yes/no
If yes use ~/.lmod.d/.cache, if no use
~/.lmod.d/__cache__
There was a problem hiding this comment.
this is just a default, --update-lmod-caches takes an argument if you want to update caches in another location
since .lmod.d/.cache is the default in Lmod, I think just keeping this as is makes sense
|
@pforai: OK, I'll try and tackle that bit in here, as discussed... It should boil down to always updating the cache before every build is started. Although this will take a little while, it will avoid having to scan the module tree multiple times when running, since |
…if luac is available), fix filename of compiled cache
|
Refer to this link for build results (access rights to CI server needed): |
|
@pforai: I took a look, and there's more needed to make sure we can not ignore the cache... We need to make sure that we can/should update the cache actively being used by Lmod in the environment we're running in. This could be a system cache (where we may not have permissions to to update it), or just our local user cache. So, it seems like we need the following cmdline options:
Iff Auto-updating the 'live' Lmod cache is probably also not a good idea in a general sense, since we don't know for which In short, I need to give this more thought, so we should probably stay away from also tackling that in this PR. |
|
@stdweird: please re-review? |
|
Refer to this link for build results (access rights to CI server needed): |
|
@pforai: One thing we can do is generate our own Lmod cache in |
…rect naming of 'old' compiled cache, only build moduleT cache, restructure new code a bit
|
Refer to this link for build results (access rights to CI server needed): |
|
This looks O.K. to me. You might want to add support for -D with the spider command as a way to track down Lmod problems when building the spider cache. |
easybuild/tools/options.py
Outdated
| 'optarch': ("Set architecture optimization, overriding native architecture optimizations", | ||
| None, 'store', None), | ||
| 'umask': ("umask to use (e.g. '022'); non-user write permissions on install directories are removed", | ||
| None, 'store', None), | ||
| 'update-lmod-cache': ("Update Lmod cache files in specified dir after generating module file", |
There was a problem hiding this comment.
(again) module tool neutral option name
There was a problem hiding this comment.
OK, I'll make this more neutral, something like update-modules-tool-cache.
|
the script createSystemCacheFile.sh is part of the standard Lmod distribution. But it has to be modified to be used at each site. |
|
@rtmclay: is there any reason why there is no dedicated option in Lmod to update the cache(s) at a specific location? |
|
@rtmclay what needs customisation? it would be better if EB can just use it instead trying to reproduce it in python (it would be even better if lmod had a |
|
@stdweird: the location of the cache and timestamp files, potentially also the But, I agree that there should be a dedicated cmdline option in Lmod for doing this. The only reason I'm reproducing it now is because there isn't (yet). If there was, I would use it for sure. |
|
The script createSystemCacheFile.sh can be modified to take arguments but having the lmod command do it is just wrong. I'm not going to take bug reports that updating the system cache fails when the user doesn't have permission to write to that directory. I can modify it when I get back. Or I'll take a PR on that. |
…he Lmod output of '-D avail'
|
Refer to this link for build results (access rights to CI server needed): |
|
Jenkins: test this please |
|
Refer to this link for build results (access rights to CI server needed): |
|
Jenkins: test this please |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
Conflicts: easybuild/framework/easyblock.py easybuild/tools/modules.py easybuild/tools/options.py test/framework/modulestool.py
|
Refer to this link for build results (access rights to CI server needed): |
|
@boegel Relevance? |
@rtmclay: it would be great if you could give this a review, especially w.r.t. the FIXME I have in there