add support for a parsable output of available modules

[shahzebsiddiqui]

Hello,

Not sure if environment-modules has an equivalent to spider command from Lmod https://lmod.readthedocs.io/en/latest/136_spider.html

This command is required by buildtest (https://buildtest.readthedocs.io/en/devel/) to do Software Stack Operation some of which is highlighted in the following:

• https://buildtest.readthedocs.io/en/devel/concepts.html
• https://buildtest.readthedocs.io/en/devel/module_operation.html

The most important of them being to automate module load testing which buildtest can do already see https://buildtest.readthedocs.io/en/devel/module_operation.html#module-load-testing-buildtest-module-loadtest

If envrionment-modules can support this feature, I can add support in buildtest to support environment-modules. Take a look at my recent talk in SC19 at https://hust-workshop.github.io/2019_presentations/1-buildtest.pdf that highlights the module operations buidltest can perform. Lmod spider is the only sane way of figuring out how to load modules and its dependencies so buildtest requires Lmod in the system.

If it is within scope of this project to support a feature like this please formulate a json like structure that provides details on all modules so I can utilize it in buildtest. If this is outside scope , please disregard this request. Since some HPC facilities are using environment-modules instead of Lmod i am unable to use buildtest to test their software stack.

[xdelaruelle]

Hello Shahzeb,

From my understanding what you need is the list of all available modules in a parsable way. Best that can be achieved currently is to parse the output of a module avail -t and filtering out modulepath and module symbols. Like it is done in completion script .

I will introduce a new command-line option --json to get output in a json format for avail and whatis commands.

Regards,
Xavier

[shahzebsiddiqui]

Hi @xdelaruelle ,

So i just ran this and it looks like this feature requires bit of work before i can do anything with this.

ssi29@hansolosim> module avail -t
/mxg-hpc/users/ssi29/easybuild-HMNS/modules/all/Core/:
/mxg-hpc/users/ssi29/spack/modules/linux-rhel7-x86_64/Core/:
/mxg-hpc/users/ssi29/easybuild/modules/all/:
/usr/share/Modules/modulefiles:
dot
module-cvs
module-info
modules
null
use.own
/etc/modulefiles:
ssi29@hansolosim> module av

--------------------------------------------------------------------------------------- /usr/share/Modules/modulefiles ----------------------------------------------------------------------------------------
dot         module-cvs  module-info modules     null        use.own

It prints the modulepath and prints all modules afterwards.

Kindly look at the spider documentation https://lmod.readthedocs.io/en/latest/136_spider.html for some ideas on how json structure is setup in Lmod. I realized after tinkering with module loads that using spider is the way to go specially when it comes to HierarchicalModules but also the fact that module names can change.

The module tool should be able to update its database if module file changes. For instance spack modules use hash at end of module https://spack.readthedocs.io/en/latest/module_file_support.html that can be changed which regenerates modules. The spider tool has fullName and parentAA key that can be used to virtually load any module. The parent modules defined by parentAA is used in Hierarchical Modules that generally require loading compilers, mpi modules before loading application module.

I hope you can try to mimic the json structure as close as possible to spider. The most important information I need is how to load the full module canonical name in environment modules and how it deals with parent modules.

Once i have this information, i'd try to implement this feature in buildtest, most important being able to automate module load test for entire tree. It's not urgent but do keep me posted once this is available.

[shahzebsiddiqui]

I am going to cross reference this issue with buildtesters/buildtest#128

Feel free to comment once you have something in place.

[xdelaruelle]

Json structure cannot be identical to the one provided by spider, as this command includes concepts that are very specific to Lmod.

What I indent to do here is to add an ability to output the result of environment query commands like avail or whatis in a parsable (json) format. It will provide the names of all available modules in enabled modulepaths. From there buildtest could attempt its load tests over these available names.

[shahzebsiddiqui]

Yes I understand, some keys wouldn't apply. So long as the json structure is source of truth in order to load any module irrespective of its module naming scheme. Should atleast support Flat Naming Scheme, and Hierarchical Modules since they are most commonly used.

The intent for this feature request is to let the module tool know the exact order of how modules are to be loaded, if this information can be parsed through json structure that would solve my problem. When designing this feature, also keep in mind that module names may change so json content should reflect the change through some update mechanism.

[xdelaruelle]

A --json command line switch has been added for the avail, list, whatis, search and savelist sub-commands. When set output is produced in a JSON format.

[shahzebsiddiqui]

thanks @xdelaruelle i will check it out and see how it can be integrated with buildtest.