feature : customization of modules from configuration files #744

alalazo · 2016-04-05T12:03:29Z

Modifications :

during module file generation all the dependencies can add variables to run_env (previously it was only extendee)
it is possible to filter modifications to specific environment variables out of module files
autoload and/or prereq in tcl module files
autoload and/or prereq in dotkit module files
it is possible to customize the environment modifications in modules
it is possible to conditionally blacklist modules
it is possible to use a naming scheme like in Modules in directories and fixes for setup-env.csh #498 and to issue conflicts in tcl modules
regression tests on all the bugs found by @glennpj

Example : filter modifications out of module files

Modifications to certain environment variables in module files are generated by default. Suppose you would like to avoid having CPATH and LIBRARY_PATH modified by your dotkit modules. Then a user modules.yaml like:

modules:
  dotkit:
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']  # Exclude changes to any of these variables

will generate dotkit module files that will not contain modifications to either CPATH or LIBRARY_PATH and tcl module files that instead will contain those modifications.

Example : autoload direct dependencies in `tcl` module files

The following lines in modules.yaml:

modules:
  tcl:
    all:
      autoload: 'direct'

will produce tcl module files that will automatically load their direct dependencies. Adding prerequisites could be done in a similar way :

modules:
  tcl:
    all:
      prerequisites: 'direct'

Example : customize module file entries

It's possible to customize the entries written in module files. For instance with:

modules:
  tcl:
    all:
      environment:
        set: ['BAR,bar']
    ^openmpi:: # Note the double ':'
      environment:
        set: ['BAR,baz']
    zlib:
      environment:
        set: ['FOO,foo']
    zlib%gcc@4.8:
      environment:
        set: ['FOOBAR,foobar']

what will happen is :

every module will have a setenv BAR bar line
unless the module depends on openmpi in which case setenv BAR baz ('::' overrides previous rules, to be consistent with what happens for different configuration sections)
anything that matches zlib will have setenv FOO foo
anything that matches zlib%gcc@4.8 will have setenv FOOBAR foobar

Example : don't generate modules for things that are built with the system compiler

The following configuration file :

modules:
  tcl:
    whitelist: ['gcc', 'llvm']  # whitelist will have precedence over blacklist
    blacklist: ['%gcc@4.4.7']

will skip module file generation for anything that satisfies %gcc@4.4.7, with the exception of gcc and llvm.

Example : customize the naming scheme and insert conflicts

A configuration file like:

modules:
  tcl:
    naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
    all:
      conflict: ['{name}', 'intel/14.0.1']

will create module files that will conflict with intel/14.0.1 and with the base directory of the same module (i.e. you cannot have two versions of the same things loaded at the same time). The conflict directive currently accepts as items :

string literals
string "formats" that match part of the dirname in naming_scheme

…t, not only extendees

… KeyError

modules : added skeleton to permit modifications based on specs

alalazo · 2016-04-06T15:37:22Z

lib/spack/spack/modules.py

-                pass
+        # TODO : the code down below is quite similar to build_environment.setup_package and needs to be
+        # TODO : factored out to a single place
+        for item in dependencies(self.spec, 'All'):


@tgamblin Adding these three lines:

for item in self.spec.traverse(order='post', depth=True, cover='nodes', root=False): tty.msg(item) tty.msg('')

immediately above here gives to me the following output for gcc@5.3.0 during spack module refresh:

==> (1, binutils@2.26%gcc@4.8+gold~krellpatch~libiberty=production) ==> (1, gmp@6.1.0%gcc@4.8=production) ==> (2, gmp@6.1.0%gcc@4.8=production) ==> (1, isl@0.14%gcc@4.8=production^gmp@6.1.0%gcc@4.8=production) ==> (2, gmp@6.1.0%gcc@4.8=production) ==> (3, gmp@6.1.0%gcc@4.8=production) ==> (2, mpfr@3.1.4%gcc@4.8=production^gmp@6.1.0%gcc@4.8=production) ==> (1, mpc@1.0.3%gcc@4.8=production^gmp@6.1.0%gcc@4.8=production^mpfr@3.1.4%gcc@4.8=production) ==> (2, gmp@6.1.0%gcc@4.8=production) ==> (1, mpfr@3.1.4%gcc@4.8=production^gmp@6.1.0%gcc@4.8=production)

The point that I want you to note is that we are visiting gmp@6.1.0%gcc@4.8=production multiple times. The behavior won't change if you remove depth=True and is not consistent with what happens at build time (where the nodes are visited only once).

I really don't get if we miss a call to some method before entering here (to collapse nodes that refers to the same spec) or if it is the way this is supposed to work.

alalazo · 2016-04-06T15:42:28Z

@tgamblin @citibeth @adamjstewart @glennpj @nrichart (or anybody else that might be interested in this), I think I am ready to collect a first round of comments / questions on this PR. Fire at will 😄

alalazo · 2016-04-06T15:44:16Z

lib/spack/spack/modules.py

+                       '}}')
+
+    prerequisite_format = 'prereq {module_file}\n'
+


@tgamblin Do you know if you can do something similar to autoload_format and prerequisite_format with dotkit?

adamjstewart · 2016-04-06T16:15:50Z

This looks very useful. We currently use SoftEnv at Argonne, but are planning to switch to some form of modules within the next year. Not sure which are currently supported by Spack or the pros and cons of each, but that would obviously influence our decision.

Questions:

Are these module configurations going to be part of the main Spack repo, or will new users need to figure everything out for themselves? It would be great to keep them in Spack, so I can install and immediately load them with all the proper environment variables set.
Will Spack be able to use these environment settings when trying to install other packages? The problems we were having with the Intel compilers comes to mind. Many of those problems can be solved by having the right library paths set by Spack.

luigi-calori · 2016-04-06T16:21:41Z

This work sound really interesting for our case.
Looking at the code, It seems one can select automatic loading or requiring dependent modules .
I would like to test and see what else could be missing in order to produce modules similar to what we currently (hand) edit. ( header and footer configuration likely)
Is it ready to test? Can I just try to merge it with current develop branch?
Thanks for the work

alalazo · 2016-04-06T21:41:59Z

@adamjstewart

This looks very useful. We currently use SoftEnv at Argonne, but are planning to switch to some form of modules within the next year. Not sure which are currently supported by Spack or the pros and cons of each, but that would obviously influence our decision.

Currently develop supports tcl and dotkit, PR #107 adds support for lmod.

Are these module configurations going to be part of the main Spack repo, or will new users need to figure everything out for themselves? It would be great to keep them in Spack, so I can install and immediately load them with all the proper environment variables set.

The features above are meant for user level configuration files while the one coming with spack will just enable the creation of both tcl and dotkit module files (without any customization). Anyhow, spack uses sensible lookup rules and the information provided by dependencies to create the default module files. In my experience this is what you want 99% of the time.

The customization covers some particular needs. For instance, at my site we don't want to have CPATH set in module files (whereas spack default is to set it). We'll solve the issue with a modules.yaml like:

modules::
  enable: ['tcl']
  tcl:
    all:
      filter:
        environment_blacklist: ['CPATH']

Another case is IntelMPI, where the fabrics in use are set at run-time using environment variables. You can add these using something like:

modules::
  enable: ['tcl']
  tcl:
    intelmpi=<your-architecture>:
      environment:
        set : [<set-the-environment-variables-here>]

that will modify only modules matching intelmpi=<your-architecture>.

Will Spack be able to use these environment settings when trying to install other packages? The problems we were having with the Intel compilers comes to mind. Many of those problems can be solved by having the right library paths set by Spack.

Not really, this PR is just meant to address module files creation. I wonder in fact if there are cases where you need per-site custom information during the installation phase (I can't think of any right now). The point with Intel compiler is a good one, but then I think that compilers are a bit different from normal packages in the sense that you don't depend on a compiler, and so far the mechanism used to inject customization in a package relies on traversing its dependencies.

alalazo · 2016-04-06T21:48:35Z

@citibeth I forgot to mention that this:

modules::
  enable: ['tcl']
  tcl:
    ^python:
      autoload: 'all'

will be quite close to what you want to achieve with #721 : all the modules of packages that depend on python will autoload all their dependencies.

glennpj · 2016-04-06T22:48:27Z

@alalazo This looks awesome at a high level. The flexibility of working with subsets of packages and dependencies will give spack features that other systems do not have, that I know of anyway.

@adamjstewart If you are looking to use an environment module system then I recommend Lmod. It can do everything that TCL modules can do plus more. It works with TCL module files through a converter so it works right now in spack.

…/custom_modulefile_from_config

glennpj · 2016-04-09T19:14:07Z

@alalazo I pulled the PR and tried a spack module refresh after setting the following in modules.yaml:

tcl:
    all:
      autoload: direct

The modules that set any module loading are non-functional with Lmod.

module show python-2.7.11-gcc-5.3.0-zowdyyxtbfjheerrhpvewuri34noptxf 
Lmod has detected the following error: 
/home/gjohnson/spack/share/spack/modules/linux-x86_64/python-2.7.11-gcc-5.3.0-zowdyyxtbfjheerrhpvewuri34noptxf:
(python-2.7.11-gcc-5.3.0-zowdyyxtbfjheerrhpvewuri34noptxf): extra characters after close-brace

Looking a the module file itself, there are no line breaks for the autoload statements. The following is all on one line:

if ![ is-loaded zlib-1.2.8-gcc-5.3.0-3cdrf2tpqqmo3qdo5hhjitio62iqgxed ] {    puts stderr "Autoloading zlib-1.2.8-gcc-5.3.0-3cdrf2tpqqmo3qdo5hhjitio62iqgxed"    module load zlib-1.2.8-gcc-5.3.0-3cdrf2tpqqmo3qdo5hhjitio62iqgxed}if ![ is-loaded ncurses-6.0-gcc-5.3.0-bo3dzyuwbnbwpdbqjupopedgwbuklmnt ] {    puts stderr "Autoloading ncurses-6.0-gcc-5.3.0-bo3dzyuwbnbwpdbqjupopedgwbuklmnt"    module load ncurses-6.0-gcc-5.3.0-bo3dzyuwbnbwpdbqjupopedgwbuklmnt}if ![ is-loaded sqlite-3.8.5-gcc-5.3.0-hhq5za5zica6omruoyrbw7g4scbzta2r ] {    puts stderr "Autoloading sqlite-3.8.5-gcc-5.3.0-hhq5za5zica6omruoyrbw7g4scbzta2r"    module load sqlite-3.8.5-gcc-5.3.0-hhq5za5zica6omruoyrbw7g4scbzta2r}if ![ is-loaded readline-6.3-gcc-5.3.0-752uupejxmzzzroligqszrejh74bs5r7 ] {    puts stderr "Autoloading readline-6.3-gcc-5.3.0-752uupejxmzzzroligqszrejh74bs5r7"    module load readline-6.3-gcc-5.3.0-752uupejxmzzzroligqszrejh74bs5r7}if ![ is-loaded openssl-1.0.2g-gcc-5.3.0-zg6cjmcacvleczjal5b2ktffwhfkxp4c ] {    puts stderr "Autoloading openssl-1.0.2g-gcc-5.3.0-zg6cjmcacvleczjal5b2ktffwhfkxp4c"    module load openssl-1.0.2g-gcc-5.3.0-zg6cjmcacvleczjal5b2ktffwhfkxp4c}if ![ is-loaded bzip2-1.0.6-gcc-5.3.0-alatbh7cqp2aobgcvg6zxrasrmfsdfdk ] {    puts stderr "Autoloading bzip2-1.0.6-gcc-5.3.0-alatbh7cqp2aobgcvg6zxrasrmfsdfdk"    module load bzip2-1.0.6-gcc-5.3.0-alatbh7cqp2aobgcvg6zxrasrmfsdfdk}prepend-path PATH "/home/gjohnson/spack/opt/spack/linux-x86_64/gcc-5.3.0/python-2.7.11-zowdyyxtbfjheerrhpvewuri34noptxf/bin"

glennpj · 2016-04-09T19:24:12Z

I wonder if considering module naming should be part of this. I am referring to the actual names of the modules and not the layout. As mentioned in PR #498 the hash in the name is not human parseable. Since an environment module system is ultimately the interface end users have to the installed software the names need to be "friendly". If doing autoloading and prereqs then those names need to be listed in dependent module files.

alalazo · 2016-04-10T09:15:41Z

@glennpj I still have to work the details of that, but as far as I can tell module 'layout' and module 'naming' should be the same thing. Currently the core part in constructing a module name is the function use_name, which returns something like:

return "%s-%s-%s-%s-%s" % (
    self.spec.name, self.spec.version,
    self.spec.compiler.name,
    self.spec.compiler.version,
    self.spec.dag_hash())

The idea I have in mind is to specify the various components from modules.yaml and being able to substitute the dash with a slash. An example along the line of #498 :

modules:
  tcl:
    naming: '{name}/{version}-{compiler-name}-{compiler-version}'

The things I still have to think about are basically how to handle file name collisions and how to deal with variants (e.g. you may want to have the concrete mpi in the name too, or something like that).

For collisions I believe that writing the full spec in a comment within the module file, and ask the user for permission to overwrite an existing module file will solve the problem. For variants I guess I'll need to give a try to some options and see which one seems to be the best. Does that make sense to you?

Regarding making changes to the naming scheme part of this PR, I would rather finalize what is already in place and submit another PR once #744 has been merged. What do you think @tgamblin ?

alalazo · 2016-04-10T09:20:19Z

@glennpj Thanks for the bug report, I forgot the newlines when refactoring the multi-line strings 😄 I'll fix them asap

alalazo · 2016-04-10T11:33:00Z

@glennpj Should be fixed now.

coveralls · 2016-05-10T15:32:01Z

Coverage increased (+1.4%) to 64.586% when pulling 71e49e2 on epfl-scitas:custom_modulefiles_from_config into 1a563c2 on LLNL:develop.

alalazo · 2016-05-10T15:37:34Z

@tgamblin The two changes are in. Now I miss docs : would you prefer to have them in a separate PR?

tgamblin · 2016-05-10T15:40:30Z

@alalazo: same PR is ok with me!

…/custom_modulefile_from_config

glennpj · 2016-05-11T02:55:06Z

@alalazo Would it be possible to add the ability to have module load directives specified in the modules.yaml file as described in #744 (comment)?

Thanks.

tgamblin · 2016-05-11T03:04:08Z

@glennpj: for the example in the comment, I believe @alalazo's prereq/autoload support does what you want. Are you looking to also add certain system modules as prereqs? I think that would be useful but maybe it could be added in another PR?

glennpj · 2016-05-11T03:25:19Z

@tgamblin What I am referring to is not covered by autoloading. Being able to add explicit load directives in the modules.yaml file would be useful for loading modules that are not built with spack. Spack does not know about them but the admins/users do.

It would also be useful when using extension activation. For example, py-numpy depends on openblas so will autoload the openblas module. However, if py-numpy is activated then the py-numpy module does not need to be loaded, just the python module does. However, the openblas module would not be autoloaded when the python module is loaded. This would definitely be a new PR to handle activation but being able to add those in modules.yaml would be a work-around.

I am more interested in the external, non-spack built, module case but it could certainly be another PR.

tgamblin · 2016-05-11T03:34:37Z

@glennpj:

For example, py-numpy depends on openblas so will autoload the openblas module. However, if py-numpy is activated then the py-numpy module does not need to be loaded, just the python module does.

This one actually sounds like a bug -- the numpy binaries should have RPATHs for their dependencies and you shouldn't need to load openblas. In the activation case the goal is for things to "just work".

For the non-activated case, the dep management here should handle loading anything that needs a PYTHONPATH as well as other packages (though again they should be RPATH'd).

Either way, though, I see lots of clear use cases for the external, non-spack-built case. So I think that should go in eventually. I can't implement it at the moment though so up to @alalazo whether he wants to put it in here or leave it for another PR.

citibeth · 2016-05-11T03:42:48Z

On Tue, May 10, 2016 at 11:34 PM, Todd Gamblin notifications@github.com
wrote:

@glennpj https://github.com/glennpj:

For example, py-numpy depends on openblas so will autoload the openblas
module. However, if py-numpy is activated then the py-numpy module does not
need to be loaded, just the python module does.

This one actually sounds like a bug -- the numpy binaries should have
RPATHs for their dependencies and you shouldn't need to load openblas.

I agree, it's a bug, Python extensions don't currently have RPATHs. See:

#935

I believe fixing the bug involves figuring out the right incantations with
Python's setuptools to get RPATH included.

-- Elizabeth

glennpj · 2016-05-11T03:54:32Z

@citibeth I believe you meant issue #719.

@tgamblin Yes, the non-activated case is fine. Even with RPATH module loading is handy when a package needs environment variables of it dependencies to be set.

This reverts commit 71e49e2.

…/custom_modulefile_from_config Conflicts: lib/spack/spack/config.py

alalazo · 2016-05-11T15:22:45Z

@tgamblin @glennpj docs are in. If you don't mind I would prefer to merge this and work on the feature request in another PR. I start to be a little bit scared by the size of this one...

tgamblin · 2016-05-11T15:24:59Z

@alalazo: I empathize with your sentiment 😄

Thanks for adding the feature for @glennpj!

tgamblin · 2016-05-11T15:26:00Z

etc/spack/modules.yaml

@@ -5,4 +5,14 @@
 # although users can override these settings in their ~/.spack/modules.yaml.
 # -------------------------------------------------------------------------
 modules:
+  prefix_inspections: {
+        bin: ['PATH'],


what's the comma at the end of the line here?

Oh I see -- the whole thing is in braces. I don't think you have to do that if the dict is spread over multiple lines.

glennpj · 2016-05-11T15:30:09Z

@alalazo Another PR is fine by me. I think this PR is awesome by the way!

Thanks.

tgamblin · 2016-05-11T16:10:01Z

Merged! Thanks to @alalazo for doing all this work!

alalazo · 2016-05-11T17:20:04Z

@glennpj @tgamblin Thank you!

alalazo added 4 commits April 5, 2016 13:20

module file : added filtering based on environment variable name

d546d82

module file : reverted to sensible default

7fc0b1e

EnvModules : narrowing the type of exceptions being "handled"

c352249

module file : all the dependencies can affect the run_time environmen…

ca7d701

…t, not only extendees

alalazo changed the title ~~[WIP] feature : customization of module files from configuration files~~ [WIP] feature : customization of modules from configuration files Apr 5, 2016

alalazo added 8 commits April 5, 2016 18:03

module file : added autoload and prereq

2968b60

modules : don't pass stream around

5da37c5

modules : iterating on all the dependencies doesn't reuire try/except…

670cb42

… KeyError

modules : more sensible name to blacklist environment variables

d636b4f

modules : added skeleton to permit modifications based on specs

modules : added environment modifications from configuration file

e993c17

modules : customization based on specs

e4cfe74

modules : all the other strings are lower case

7904f75

modules : removed debug leftover

21b7734

alalazo changed the title ~~[WIP] feature : customization of modules from configuration files~~ feature : customization of modules from configuration files Apr 6, 2016

alalazo reviewed Apr 6, 2016
View reviewed changes

alalazo added 2 commits April 7, 2016 10:48

modules : added stub for dotkit (to be verified)

7dc1524

Merge branch 'develop' of https://github.com/LLNL/spack into features…

1714f41

…/custom_modulefile_from_config

tcl module file : added new-lines in autoload (per @glennpj bug report)

a5a5dbc

alalazo added 4 commits May 10, 2016 13:37

modules : prefix_inspections moved to modules.yaml

f8f71b1

modules : changed syntax for environment modifications

0b7c673

modules : added warning if a user tries to add prerequisite with dotkit

c3f3f26

flake8 : fixed all issues?

71e49e2

Merge branch 'develop' of https://github.com/LLNL/spack into features…

965e7f4

…/custom_modulefile_from_config

alalazo added 4 commits May 11, 2016 15:42

modules : added docs

b215b19

Revert "flake8 : fixed all issues?"

22bb056

This reverts commit 71e49e2.

Merge branch 'develop' of https://github.com/LLNL/spack into features…

78ae5d7

…/custom_modulefile_from_config Conflicts: lib/spack/spack/config.py

flake 8 : fixed checks

bb4b6c8

tgamblin reviewed May 11, 2016
View reviewed changes

tgamblin merged commit bb4b6c8 into spack:develop May 11, 2016

alalazo deleted the custom_modulefiles_from_config branch May 11, 2016 17:20

tgamblin mentioned this pull request May 12, 2016

Allow setting of LD_LIBRARY_PATH for compilers #943

Closed

alalazo mentioned this pull request May 23, 2016

module files : explicit load and mnemonic suffixes #984

Merged

4 tasks

matz-e pushed a commit to matz-e/spack that referenced this pull request Apr 27, 2020

bump entity-management to v1.1.3 (spack#744)

8961e8c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature : customization of modules from configuration files #744

feature : customization of modules from configuration files #744

alalazo commented Apr 5, 2016 •

edited

Loading

alalazo Apr 6, 2016

alalazo commented Apr 6, 2016

alalazo Apr 6, 2016

adamjstewart commented Apr 6, 2016

luigi-calori commented Apr 6, 2016

alalazo commented Apr 6, 2016

alalazo commented Apr 6, 2016

glennpj commented Apr 6, 2016

glennpj commented Apr 9, 2016

glennpj commented Apr 9, 2016

alalazo commented Apr 10, 2016

alalazo commented Apr 10, 2016

alalazo commented Apr 10, 2016

coveralls commented May 10, 2016 •

edited

Loading

alalazo commented May 10, 2016

tgamblin commented May 10, 2016

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

citibeth commented May 11, 2016

glennpj commented May 11, 2016

alalazo commented May 11, 2016

tgamblin commented May 11, 2016

tgamblin May 11, 2016

tgamblin May 11, 2016

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

alalazo commented May 11, 2016

feature : customization of modules from configuration files #744

feature : customization of modules from configuration files #744

Conversation

alalazo commented Apr 5, 2016 • edited Loading

Modifications :

Example : filter modifications out of module files

Example : autoload direct dependencies in tcl module files

Example : customize module file entries

Example : don't generate modules for things that are built with the system compiler

Example : customize the naming scheme and insert conflicts

alalazo Apr 6, 2016

Choose a reason for hiding this comment

alalazo commented Apr 6, 2016

alalazo Apr 6, 2016

Choose a reason for hiding this comment

adamjstewart commented Apr 6, 2016

luigi-calori commented Apr 6, 2016

alalazo commented Apr 6, 2016

alalazo commented Apr 6, 2016

glennpj commented Apr 6, 2016

glennpj commented Apr 9, 2016

glennpj commented Apr 9, 2016

alalazo commented Apr 10, 2016

alalazo commented Apr 10, 2016

alalazo commented Apr 10, 2016

coveralls commented May 10, 2016 • edited Loading

alalazo commented May 10, 2016

tgamblin commented May 10, 2016

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

citibeth commented May 11, 2016

glennpj commented May 11, 2016

alalazo commented May 11, 2016

tgamblin commented May 11, 2016

tgamblin May 11, 2016

Choose a reason for hiding this comment

tgamblin May 11, 2016

Choose a reason for hiding this comment

glennpj commented May 11, 2016

tgamblin commented May 11, 2016

alalazo commented May 11, 2016

alalazo commented Apr 5, 2016 •

edited

Loading

Example : autoload direct dependencies in `tcl` module files

coveralls commented May 10, 2016 •

edited

Loading