New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Is it possible to query if a package from a family is loaded? #114

Closed
alalazo opened this Issue Apr 20, 2016 · 8 comments

Comments

Projects
None yet
2 participants
@alalazo

alalazo commented Apr 20, 2016

I am currently trying to implement something like :

if isloaded(<generic-family-name>) then
  name, version = <get-concrete-name-of-the-package-providing-the-service>
  -- Path manipulation based on `name` and `version`
end

within lua module files. The idea is to add other "services" with multiple providers (e.g. "lapack") side by side with MPI. Is this possible given the current state of the art in Lmod?

@rtmclay

This comment has been minimized.

Member

rtmclay commented Apr 20, 2016

Lmod doesn't know which module has set the family directive. If you use the software hierarchy then you can use the hierarchyA function.

You can read more about its use here: https://sourceforge.net/p/lmod/mailman/message/32375349/

@rtmclay

This comment has been minimized.

Member

rtmclay commented Apr 20, 2016

BTW. You might think that you can set env. vars. in the compiler or mpi modulefile to be used in dependent modules (say boost or fftw). When I have tried this I have run into trouble when unloading. Typically the compiler or mpi modulefile is unloaded first then the dependent module. If the dependent module needs the value of this env. var. to correctly unload the variable wont have a value.

@alalazo

This comment has been minimized.

alalazo commented Apr 20, 2016

Thank you very much for the prompt reply!

I'll try the few tips you have given and, if you are interested, I'll post here the results. FYI : I am using the "typical" hierarchical layout of module files (Core, Compiler, MPI) and am trying to extend it beyond these three components.

@rtmclay

This comment has been minimized.

Member

rtmclay commented Apr 20, 2016

Please let us know what you find. It is a tough but important problem.

On Wednesday, April 20, 2016, Massimiliano Culpo notifications@github.com
wrote:

Thank you very much for the prompt reply!

I'll try the few tips you have given and, if you are interested, I'll post
here the results. FYI : I am using the "typical" hierarchical layout of
module files (Core, Compiler, MPI) and am trying to extend it beyond these
three components.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#114 (comment)

Robert McLay, Ph.D.
TACC
Manager, HPC Software Tools
(512) 232-8104

@alalazo

This comment has been minimized.

alalazo commented Apr 25, 2016

Hi, following your advice (use environment variables), I was able to generate a hierarchy of directories that treats lapack the same way MPI is treated. For instance :

$ module av

--------------------------------------------------- /home/mculpo/PycharmProjects/spack/share/spack/lmod/production/netlib-lapack/3.6.0/openmpi/1.10.2/gcc/5.3.0 ---------------------------------------------------
   netlib-scalapack/2.0.2-lkqbcg (L)

------------------------------------------------------------- /home/mculpo/PycharmProjects/spack/share/spack/lmod/production/openmpi/1.10.2/gcc/5.3.0 -------------------------------------------------------------
   fftw/3.3.4-edzq54    hdf5/1.10.0-vahw2e

-------------------------------------------------------------------- /home/mculpo/PycharmProjects/spack/share/spack/lmod/production/gcc/5.3.0 ---------------------------------------------------------------------
   binutils/2.26-lbtqaa    fftw/3.3.4-44hx73          (D)    libsigsegv/2.10-hqenri    netlib-lapack/3.6.0-6rmmoh (L)    py-nose/1.3.7-cvdfyu           sqlite/3.8.5-vdlp2t
   bison/3.0.4-u6u3a3      flex/2.5.39-5uz3jb                libtool/2.4.6-bi2dim      openblas/0.2.17-ad7cge            py-setuptools/20.7.0-w5b4h3    szip/2.1-a3rdlo
   bzip2/1.0.6-4jhgi7      hwloc/1.11.2-c2njd5               m4/1.4.17-ewlgb6          openmpi/1.10.2-242gmm      (L)    python/2.7.11-j3kgqn           zlib/1.2.8-5jh2uc
   cmake/3.5.1-cervyp      libpciaccess/0.13.4-svfuqi        ncurses/6.0-j2xtwk        openssl/1.0.2g-xezdlh             readline/6.3-msqzt5

----------------------------------------------------------------------- /home/mculpo/PycharmProjects/spack/share/spack/lmod/production/Core -----------------------------------------------------------------------
   cmake/3.5.0-du4emo    cmake/3.5.1-3hldns (D)    gcc/5.3.0-n2e5zs (L)    llvm/3.8.0-xcsjjw

------------------------------------------------------------------------------------------- /usr/share/modules/versions -------------------------------------------------------------------------------------------
   3.2.10

----------------------------------------------------------------------------------------- /usr/share/modules/modulefiles ------------------------------------------------------------------------------------------
   dot    module-git    module-info    modules    null    use.own

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

$ module list

Currently Loaded Modules:
  1) gcc/5.3.0-n2e5zs   2) openmpi/1.10.2-242gmm   3) netlib-lapack/3.6.0-6rmmoh   4) netlib-scalapack/2.0.2-lkqbcg

$ module load openblas

Lmod is automatically replacing "netlib-lapack/3.6.0-6rmmoh" with "openblas/0.2.17-ad7cge"


The following have been reloaded with a version change:
  1) netlib-scalapack/2.0.2-lkqbcg => netlib-scalapack/2.0.2-5rbiap

the correct scalapack gets reloaded on a lapack reload. Lmod seems to be already able to handle this situation (seriously, this is a great tool!) with the exception of module spider.

Unfortunately I don't know lua at all but, if I can make a wild guess, I would say that at the moment spider can't handle conditional modifications to module paths, something like (taken from the openblas module):

local mpi_name = os.getenv("LMOD_MPI_NAME")
local mpi_version = os.getenv("LMOD_MPI_VERSION")

if mpi_name then
  local t = pathJoin("/home/mculpo/PycharmProjects/spack/share/spack/lmod/production", "openblas/0.2.17", mpi_name, mpi_version, "gcc/5.3.0")
  prepend_path("MODULEPATH", t)
end

This command finds netlib-scalapack only if an mpi module (setting the LMOD_MPI_* environment variables) has already been loaded, while I would expect to tell me since the beginning that I need to load a compiler, an mpi and a lapack to have it. If it may be of use I can provide a tgz of the module files.

@rtmclay

This comment has been minimized.

Member

rtmclay commented Apr 25, 2016

Yes, and it isn't going to be able to either. Also make sure that you don't have the use.own module in your modulepath.

The spider cache file is built once and can't know about conditional modifications to the modulepath.

R.

@alalazo

This comment has been minimized.

alalazo commented Apr 25, 2016

I was wondering, did you already try something like what I am doing above (I mean : having multiple providers for different and orthogonal services at the same time)? Would you be interested in modification of Lmod that could take care of this case when "spidering" for modules? If so I may try to code such modifications, but may need some guidance in the process.

BTW, I am using a customization of spack to generate the modulefiles. In case you are interested in the thing you may find the source code here

@rtmclay

This comment has been minimized.

Member

rtmclay commented Apr 25, 2016

I'll be happy to answer any questions you have. If you do, you should move them to the Lmod mailing list as more people read that.

@alalazo alalazo referenced this issue May 12, 2016

Closed

lmod : support for hierarchical modules #107

3 of 3 tasks complete

@rtmclay rtmclay closed this Jul 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment