Too many calls to "module use ..."

[ocaisa]

During a dry run of Qt I did a debug log and counted over 1100 calls to module use with a HMNS...and the eb command took 3.5 minutes to complete. With a flat naming scheme it was still almost 700 calls and 2.7 minutes. I'm pretty sure that this is because module use is pretty expensive for lmod.

Can we reduce the number of calls? Or configure our lmod use so that it's not so expensive? Thoughts @boegel @wpoely86 @rtmclay ?

[wpoely86]

Yeah, I see exactly the same: building Perl (+extensions) takes 4-5 hours because of excessive calls to Lmod.

[boegel]

The stats collected in #946 (comment) help with confirming that there's way too many use calls...

I hope to chase this down really soon.

[pescobar]

I have also experienced this issue. When having many modules just launching easybuild to install a new easyconfig takes minutes.

[boegel]

OK, I just spent some time on hunting this down. Here's the bottom line:

• module use is mainly triggered by check_module_path in ModulesTool, which is called every time an instance of ModulesTool is created (usually via the modules_tool function); this happens quite often , which explains the ~2.5k calls to --version and the 10k calls to use (cfr. collect avg timings for module subcmds #946 (comment))
• ModulesTool used to be a singleton class, for reasons closely related to this (module commands being run over and over again without good reason), cfr. make ModulesTool a singleton, use 'module use/unuse' rather than fiddling with $MODULEPATH, use $LMOD_QUIET rather than $LMOD_EXPERT #944
• however, we had to stop doing that because of bugs, cfr. stop making ModulesTool class a singleton #1299; this change has a far bigger impact on performance than anticipated, clearly...

Possible solutions:

• cache output of --version (since it's not going to change)
• avoid calling modules_tool over and over again; exactly once per installation and passing around the instance should be sufficient
• avoid blindly calling use over and over again to make sure $MODULEPATH is what it should be; at the very least, we should check whether $MODULEPATH is already what it should be, and just skip the use calls in case it is (it's likely the skipping can be done quite often)

The combination of all of the above should result in a fix for the reported issue...

I'll look into this, hopefully very soon.

[boegel]

should be fixed by #1735
@ocaisa please test?

[boegel]

fix in #1735 confirmed