json output

[wpoely86]

It would be cool if you could do something like module -o json avail. (replace json with your favorite data-interchange format). It would make parsing output of Lmod a lot easier by third party tools like Easybuild.

I see that the spider already has support of json, xml, ... so with my very limited knowledge about Lmod, I don't think this can be very hard?

[rtmclay]

Lmod like TCL/C modules and the tcl pure version supports the -t (or --terse) option. So you can do:

module -t avail
module -t list
module -t spider

I know that Easybuild is using the -t option to parse the output of all the module systems they support.

[wpoely86]

module -t spider does not seem to do anything different?

It can also cause problems with very very long paths?
We also had an issue how to be sure that a module exists (hidden or not), aka process the output of module show. Lmod doesn't indicate by return code if there is a module there or not.

[rtmclay]

What do you mean that module -t spider doesn't do anything different? Is it the same as module spider or is it the same as module -t avail?

[wpoely86]

a no, my mistake. It works as it is supposed to do. It's module -t spider <name> that doesn't do anything differen then without -t.

[rtmclay]

I have fixed "module show " to set the error core if the module doesn't exist. I am looking at "module -t spider " issue. The new version is pushed to the HEAD.

[rtmclay]

Fixed module -t spider <name> and pushed to repos. Note that module -t spider hdf5 will report terse output but module -t spider hdf5/1.8.13 does not report terse output. It doesn't make sense as far as I can see.

[rtmclay]

If you are using module show <name> to see if name is installed. it will work now but module -t avail should be faster at least from the Lmod point of view. This is because a show is the same as a load in term of work. And before any load the MODULEPATH is walked to find all the modules.

[wpoely86]

well, we need something to check if a module is there, even if it's hidden. I think module show <name> is our only option?

[boegel]

@wpoely86: we can only use show for hidden modules (which should be fairly rare), I'm working on that

[rtmclay]

Lmod now supports module --terse --show_hidden avail which will show all modules including the hidden ones.

[boegel]

@rtmclay: only with --terse? Also, why not --show-hidden? I've rarely seen the use of underscores in command line options...

[rtmclay]

--show_hidden works without --terse. As for underscores vs. dashes see issue #34

It is just in this case as a replacement for module show <name>. You will want to use the --terse option.

[rtmclay]

closing this issue.

[boegel]

The new --show_hidden option isn't mentioned in the release notes/README.

Am I correct when assuming it's available since v5.7.5?

[rtmclay]

You are correct. The --show_hidden option has been available since v5.7.5. I have updated the README to reflect that. I have updated HEAD.

[vsoch]

hey folks! Sorry for commenting on an old issue. It does look like -t adds "terse" that makes me press enter to read on the screen - what if I want to save a list to file (e.g., either as a list or json?) I can't find a way to do that with lmod - it looks like the other modules supports it cea-hpc/modules#303

[wpoely86]

I think that depends on your pager settings? For me it just outputs everything to the screen. My pager is set to 'less'.

[vsoch]

But should it? If I am creating a tool that needs to parse the output for a user how does relying on pager settings (where most use less) help me? Can I change it on the fly for one command?

[wpoely86]

I guess we could add a json output so that $LMOD_CMD json -t spider my-module would work.

Try it with export LMOD_REDIRECT=yes first?

[rtmclay]

Lmod checks what stderr is connected to. If it is a tty then the pager is used. If it a file then it bypasses the pager setting.

$ ml -t spider 2> spider.log

works fine.

[vsoch]

That works! However as a user I'd never guess to try that.

[vsoch]

Maybe you can advise? I'm trying to add a command to run via ssh to just get a listing of modules on a cluster. If I remember correctly the way "module" or"ml" is invoked is kind of strange, e.g., it works when I login but not when I ssh.

res = self.ssh.execute("/bin/bash -l ml -t spider > modules.txt")

or more simply from bash:

$ ssh <server> ml -t spider
bash: ml: command not found

$ ssh <server> module load -t spider
bash: module: command not found

$ ssh <server> /bin/bash -l module load -t spider
stty: standard input: Inappropriate ioctl for device

Lmod is automatically replacing "intel/19.0.4" with "gcc/10.2.1".


Due to MODULEPATH changes, the following have been reloaded:
  1) mvapich2/2.3

/bin/bash: module: No such file or directory

Could you give any pointers? Any standard library that I could at least add to the path would work! Thank you!

[rtmclay]

This is a problem with bash and how it is setup. Are you stuck with bash? You would be better off using zsh because zsh doesn't have the problems of bash. You can read about my suggestion to patch bash here: https://lmod.readthedocs.io/en/latest/030_installing.html?highlight=patching#interactive-non-login-shells

The issue is that depending on how your bash is build, it does or does not do a full interactive login when running "ssh host command" or not. That is the remote bash shell that is started on the server bypasses /etc/profile or /etc/bashrc. You should test if it starts your personal startup scripts by adding echo statements in both your ~/.profile and ~/.bashrc and do a "ssh host command" and see what happens.

If you can't patch bash or change shells to zsh then the best I can recommend is to run shell scripts on the remote host that do run a login shell so that the module command is defined.

[vsoch]

I could personally use zsh, but the average cluster user is going to have bash (and a few csh). And I'd suspect most bash installs on HPC aren't custom builds (and likely have the default bash build). Indeed with the last example (with -l to prompt a login shell) it looks like something with modules is loaded, but it still reports not being able to find the command.

That's actually a good idea - my tool has a small cache of scripts on the host, so I'll try running the command there. Will update after I try!

[vsoch]

Worked!

[vsoch]

tunel-apps/tunel@99d8bbb#diff-58840dbb8fd7c275c1be61f998c832a7a857b6ee58d6b190068b39602fa42873

Feels like a bit of a hack, but then again, so is everything in HPC, and I'm happy to have a hack over nothing at all.

Thanks for your help today!

[wpoely86]

@vsoch you could also directly call python output of Lmod?

/usr/share/lmod/lmod/libexec/lmod python -t spider some-module

[vsoch]

Oh that's interesting output! It indeed starts with the list, but also ends with a set of what looks like envars?

vtune/
vtune/latest
vtune/2016.3
vtune/2017.1
vtune/2018.0
vtune/2018.3
vtune/2019.0
vtune/2020.0
vtune/2021.2.0
xalt
xforms/
xforms/1.0.91
xsu
xsu/no.x
import os
os.environ["MODULEPATH"] = "/usr/tce/modulefiles/MPI/mvapich2/2.3:/usr/tce/modulefiles/MPI/gcc/10.2.1/mvapich2/2.3:/usr/tce/modulefiles/Compiler/gcc/10.2.1:/collab/usr/global/tools/modulefiles/toss_3_x86_64_ib/Core:/usr/tce/modulefiles/Core:/usr/apps/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core";
os.environ["_ModuleTable001_"] = "xxxxxxxxx...";
os.environ["_ModuleTable002_"] = "xxxxxxxxx...";
os.environ["_ModuleTable003_"] = "xxxxxxxxx...";
os.environ["_ModuleTable004_"] = "xxxxxxxx..";
os.environ["_ModuleTable_Sz_"] = "4";
_mlstatus = True

The risk here would be if someone didn't install lmod to that exact path (I lucked out that the node I was on did).

[wpoely86]

The python code goes to stdout, the list goes to stderr.
For the path you can use $LMOD_CMD. This should be set or else Lmod is not active (it's used in the module function itself).