Skip to content

reduce number of module load commands used to set up build environment - #5211

Draft
boegel wants to merge 5 commits into
easybuilders:developfrom
boegel:perf
Draft

reduce number of module load commands used to set up build environment#5211
boegel wants to merge 5 commits into
easybuilders:developfrom
boegel:perf

Conversation

@boegel

@boegel boegel commented Jun 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member
  • loading of modules when setting up toolchain build environment for toolchain itself (1) + dependencies (D) is not done in one go, so a single module load command rather than 1+D `module load commands;
  • build environment for installing extensions is determine once, and environment is restored right before installing an extension (without running module commands);
    • only implemented in EasyBlock.install_extensions_sequential currently;

TODO:

  • also change EasyBlock.install_extensions_parallel to restore build environment for each extension installation;
  • fix broken tests;
  • look for further "easy" wins, like avoiding module unload commands

Essentially fixes #5052

boegel added 5 commits June 21, 2026 16:57
@boegel
load list of modules in one go as opposed to running separate 'module…
5b2bbd9 
… load' commands for each of them
@boegel
refactor Toolchain._load_modules by using Toolchain._load_modules_too…
4cd0bb1 
…lchain_and_deps which loads modules for toolchain + dependencies in one go (except when --extended-dry-run is used)
@boegel
remove now unused _load_toolchain_module + _load_dependencies_modules…
531c807 
… helper methods from Toolchain class
@boegel
refactor EasyBlock.install_extensions_sequential to determine build e…
aec37a7 
…nvironment once before installing extensions, and use restore_env to restore it before installing each extension
@boegel
fix test_toy_build_hooks, no module-write hook triggered after instal…
43d3e0b 
…lation of bar extension (?!)
@boegel boegel added this to the next release (5.3.2?) milestone Jun 22, 2026
hound[bot]
hound Bot reviewed Jun 22, 2026
View reviewed changes
Comment thread easybuild/tools/toolchain/toolchain.py
else:
# first simulate loads for toolchain dependencies, if required information is available
for tc_dep in self.tcdeps:
mod_name = tcdep['short_mod_name']

@hound hound Bot Jun 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undefined name 'tcdep'

@Flamefire Flamefire mentioned this pull request Jun 23, 2026
Open
@branfosj

branfosj commented Jun 29, 2026

Copy link
Copy Markdown
Member

Installing the first 49 extensions from R-bundle-CRAN/2025.11-foss-2025 sequentially:

  • EB 4.9.4: 13:20
  • Current develop: 29:20
  • This PR: 18:29

@branfosj

branfosj commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

This reintroduces #4948

The logs below are based on an easyconfig that contains only the first extension from easybuilders/easybuild-easyconfigs#26416

The lpsymphony extension uses the values set here are fails due to a repeated mpicxx mpicxx.

Without this PR:

$ grep MPICXX /dev/shm/branfosj/tmp-up-EL8/eb-g11gy9gh/easybuild-nm4tbkom.log
== 2026-07-15 10:51:54,550 environment.py:95 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:51:54,550 environment.py:95 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:52:19,759 environment.py:95 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:52:19,759 environment.py:95 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)

With this PR - note the mpicxx mpicxx when MPICXX is defined the second time, which is when the first extension is installed:

$ grep MPICXX /dev/shm/branfosj/tmp-up-EL8/eb-odz9_d86/easybuild-6m2kgc_i.log
== 2026-07-15 10:54:40,947 environment.py:102 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:54:40,947 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:55:05,274 environment.py:102 INFO Environment variable MPICXX set to mpicxx mpicxx (previously undefined)
== 2026-07-15 10:55:05,274 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx mpicxx (previously undefined)

@boegel

boegel commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

This reintroduces #4948

The logs below are based on an easyconfig that contains only the first extension from easybuilders/easybuild-easyconfigs#26416

The lpsymphony extension uses the values set here are fails due to a repeated mpicxx mpicxx.

Without this PR:

$ grep MPICXX /dev/shm/branfosj/tmp-up-EL8/eb-g11gy9gh/easybuild-nm4tbkom.log
== 2026-07-15 10:51:54,550 environment.py:95 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:51:54,550 environment.py:95 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:52:19,759 environment.py:95 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:52:19,759 environment.py:95 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)

With this PR - note the mpicxx mpicxx when MPICXX is defined the second time, which is when the first extension is installed:

$ grep MPICXX /dev/shm/branfosj/tmp-up-EL8/eb-odz9_d86/easybuild-6m2kgc_i.log
== 2026-07-15 10:54:40,947 environment.py:102 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:54:40,947 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)
== 2026-07-15 10:55:05,274 environment.py:102 INFO Environment variable MPICXX set to mpicxx mpicxx (previously undefined)
== 2026-07-15 10:55:05,274 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx mpicxx (previously undefined)

@branfosj Was this with or without --parallel-extensions-install?

@branfosj

branfosj commented Jul 29, 2026

Copy link
Copy Markdown
Member

@branfosj Was this with or without --parallel-extensions-install?

Without -parallel-extensions-install.

With -parallel-extensions-install does not show the issue:

$ grep MPICXX /dev/shm/branfosj/tmp-up-EL8/eb-qdxer8zv/easybuild-fesbz3j1.log
== 2026-07-29 11:08:27,726 environment.py:102 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-29 11:08:27,726 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)
== 2026-07-29 11:08:39,035 environment.py:102 INFO Environment variable MPICXX set to mpicxx (previously undefined)
== 2026-07-29 11:08:39,035 environment.py:102 INFO Environment variable EBVARMPICXX set to mpicxx (previously undefined)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@hound hound[bot] hound[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

enhancement performance

Projects

None yet

Milestone

next release (5.3.2 or 5.4.0)

Development

Successfully merging this pull request may close these issues.

Building easyconfigs with many extensions is much slower with EasyBuild 5.x than with EasyBuild 4.x

2 participants

@boegel @branfosj