Skip to content

Conference call notes 20240522

Jump to bottom
Kenneth Hoste edited this page May 22, 2024 · 3 revisions

(back to Conference calls)

Notes on the 246th EasyBuild conference call, Wednesday 22 May 2024 (08:00 UTC)

Attendees

List of attendees (17):

  • Sebastian Achilles (JSC, Germany)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (TU Dresden, Germany)
  • Thomas Hayward-Schneider (Max Planck Institute for Plasma Physics (IPP))
  • Thomas Hoffman (EMBL, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Luca Marsella (CSCS, Switzerland)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Sébastien Moretti (SIB, Switzerland)
  • Alan O'Cais (CECAM + Univ. of. Barcelona, Spain)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Jure Pečar (EMBL, Heidelberg, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Alain van Hoof (TU Eindhoven, Netherlands)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.1 (3 Apr 2024)
  • next EasyBuild release: probably 4.9.2 in coming weeks, to help with transition to 5.0.0
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • ...
    • changes
      • do not run unit tests on Python 3.5 (PR #4530)
    • code cleanup
      • code cleanup in easyblock.py (PR #4519)

  • easyblocks (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • refactor GAMESS-US easyblock to directly write install.info (v2) (PR #3047)
      • add RISC-V support to Clang easyblock (PR #3325)
      • use work_dir option for run_shell_cmd instead of global chdir in custom easyblock for Go (PR #3327)
    • updates
    • changes
      • do not run unit tests on Python 3.5 (PR #3326)
    • new easyblocks

  • easyconfigs (merged PRs)

    • ~60 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • disable test that fetches from the web for torchtext extension in PyTorch-bundle v2.1.2 (PR #20484)
      • use the included gmxapi for GROMACS 2024.1 (PR #20522)
      • add new checksum for signal_1.8-0 to R-bundle-CRAN-2023.12 (PR #20527)
    • enhancements
      • add PSM2 dependency to libfabric 1.12.1 and newer (PR #20501)
    • (noteworthy) new software
      • ...
    • noteworthy software updates

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • Fix resolved (template) values in case of failure (PR #4532)
    • enhancements
      • add support for --sources-url to fetch sources from additional URLs (PR #4079)
      • Disallow template failure (PR #4516)
      • add support for data installations (PR #4474)
      • Improve download logging (PR #4518)
      • versions checks to avoid mixing major versions across the EasyBuild components (PR #4520)
      • add new get_cwd function to tools.filetools to retrieve current working directory (PR #4525)
      • Use dict.items() instead of repeatedly getting the value (PR #4533)
      • add support for easyconfig parameter module_only (PR #4537)
    • changes
      • Prefer $EBPYTHONPREFIXES over $PYTHONPATH (PR #4496)
        • open questions when using system toolchain
        • should be easy to handle in PythonPackage & PythonBundle
        • harder to deal with when set through modextrapaths
        • code that determines path to site-packages dir is quite long, unclear why...
        • complicates goal to have this enabled by default
      • add support for alternate easyconfig parameters/templates/constants (PR #4511)
        • problems with extra tests that are leaking into each other
      • Standarize behaviour on changing of working directory (PR #4525, PR #4390)
      • Make generic EasyBlock usable (WIP PR #4531)
      • More granular exit codes (PR #4534)

  • easyblocks (open PRs + issues)

    • enhancements
      • Add a sanity check for OpenFOAM's wmake to make sure it finds the compiler commands (PR #3328)
      • Always override the compiler version check for CUDA installations (PR #3329)
      • Explicitely mention that the PyTorch easyblock needs updating when failing for this reason (PR #3255)
      • add libjpeg-turbo to include dir for torchvision easyblock (PR #3322)
      • Set build type for PyTorch explicitely (PR #3332)
    • changes
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
      • set CMAKE_INSTALL_LIBDIR to lib by default (PR #3227, PR #20487)
      • Archive autotools-based QuantumESPRESSO easyblock and switch default to CMakeMake-based easyblock (PR #3306)
      • no need to use SANDCASTLE=1 in custom easyblock for PyTorch for versions >= 2.3.0, it actually causes more problems than it solves (PR #3330)
    • new
      • add generic Dataset easyblock (PR #3246)

  • easyconfigs (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • ...
    • (noteworthy) new software
      • QuantumESPRESSO:
        • 7.3.1 with cmake (PR #20138)
        • Correct configuration of QuantumESPRESSO for 6.7/7.2/7.3, move all QE easyconfigs to "legacy" easyblock (PR #20070)
      • jax 0.4.25 w/ CUDA (PR #20119)
      • RELION v5.0.0_beta.2024.0.01 w/ CUDA 12.1.1 9 (PR #19678)
      • PyTorch v2.3.0 (PR #20489)
        • Alexander is in touch with PyTorch devs on getting machine-readable output for PyTorch test suite
    • changes
      • ...

2024a common toolchains

  • GCC 13.3 as a base
  • latest version of all components (except GCC, not using 14.1):
    • foss/2024a
      • GCC 13.3 (20240521) + binutils 2.42 (20240129) (PR #20559)
      • OpenMPI 5.0.3 (20240408) (PR #17561)
        • PMIx 5.0.2 (20240321), UCX 1.16.0 (20240415), libfabric 1.21.0 (20240329)
      • FlexiBLAS 3.4.3 (20240521) + OpenBLAS 0.3.27 (20240404) + BLIS 1.0 (20240506)
      • FFTW 3.3.10 (20210914)
      • ScaLAPACK 2.2.0 (20220202)
    • intel/2024a
      • GCCcore 13.3 + binutils 2.42
      • Intel C/C++/Fortran compilers 2024.1.0 (20240327)
      • Intel MPI 2021.12.1 (20240424)
      • Intel MKL 2024.1.0 (20240327)
  • notes
    • Python 3.11.x or 3.12.x => trying with Python 3.12 for now
    • Sam: Cython should be a build dependency for Python-bundle-PyPI

Q&A / others

  • more fine-grained bundles of R extensions going forward?
    • someone should come up with a proposal on how to
  • "fat" installation of MPICH like we have with OpenMPI doesn't really seem possible
    • would need to have separate easyconfigs for different variants of MPICH, so you can opt-in to the one you need for your system
    • https://github.com/cea-hpc/wi4mpi or MPITrampoline could help, but are in some sense only a partial fix
    • some work also being done on a common ABI for MPI (but accelerator stuff is out of scope for that)
  • Sebastian: PSM2 as added to libfabric recently
Clone this wiki locally