From 488c0a786e0c64879fa0c74259dacf9eeea7593b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 26 Oct 2022 21:18:07 +0200 Subject: [PATCH 1/7] remove warning on top of 'using-easybuild' page --- docs/using-easybuild.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/docs/using-easybuild.md b/docs/using-easybuild.md index 82bd234550..c2027dbd8d 100644 --- a/docs/using-easybuild.md +++ b/docs/using-easybuild.md @@ -4,19 +4,6 @@ ## Enabling dependency resolution -!!! warning - This page will soon replace . - - ** - It still needs to be ported from *reStructuredText* (.rst) to *MarkDown* (.md), - and you can help with that! - ** - - - source: [`docs/Using_the_EasyBuild_command_line.rst` in `easybuilders/easybuild` repo](https://raw.githubusercontent.com/easybuilders/easybuild/develop/docs/Using_the_EasyBuild_command_line.rst) - - target: [`docs/using-easybuild.md` in `easybuilders/easybuild-docs` repo](https://github.com/easybuilders/easybuild-docs/tree/main/docs/using-easybuild.md) - - See for more information. - ```rst .. _using_the_easybuild_command_line: From fa2ec5394b11ac668223ce094513f17b9e44de29 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 28 Oct 2022 21:08:16 +0200 Subject: [PATCH 2/7] port using-easybuild page to MarkDown --- docs/configuration.md | 8 + docs/terminology.md | 2 + docs/using-easybuild.md | 925 ++++++++++++++++--------------- docs/writing-easyconfig-files.md | 4 + 4 files changed, 478 insertions(+), 461 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 04629be362..4d8286f056 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,5 +1,13 @@ # Configuring EasyBuild +## Supported configuration types + +### Configuration file(s) + +#### List of used configuration files + +#### Templates and constants supported in configuration files + !!! warning This page will soon replace . diff --git a/docs/terminology.md b/docs/terminology.md index 35accf8372..59a5c2e2ca 100644 --- a/docs/terminology.md +++ b/docs/terminology.md @@ -1,5 +1,7 @@ # EasyBuild terminology +## Easyconfig files + !!! warning This page will soon replace . diff --git a/docs/using-easybuild.md b/docs/using-easybuild.md index c2027dbd8d..632995d624 100644 --- a/docs/using-easybuild.md +++ b/docs/using-easybuild.md @@ -1,650 +1,653 @@ # Using the EasyBuild command line -## All available easyconfig parameters - -## Enabling dependency resolution - -```rst -.. _using_the_easybuild_command_line: - -Using the EasyBuild command line -================================ +Basic usage of EasyBuild is described in the following sections, covering the most important range of topics +if you are new to EasyBuild. -Basic usage of EasyBuild is described in the following sections, covering the most important range of topics if you are new to EasyBuild. - -``eb`` is EasyBuild’s main command line tool, to interact with the EasyBuild framework +`eb` is EasyBuild’s main command line tool, to interact with the EasyBuild framework and hereby the most common command line options are being documented. -.. _specifying_easyconfigs: -Specifying builds ------------------ +## Specifying what to install To instruct EasyBuild which software packages it should build/install and which build parameters it should use, -one or more *easyconfig files* (see :ref:`easyconfig_files`) must be specified. +one or more [*easyconfig files*](terminology.md#easyconfig-files) must be specified. This can be done in a number of ways: -* :ref:`specifying_easyconfigs_single` -* :ref:`specifying_easyconfigs_command_line` -* :ref:`specifying_easyconfigs_set_of_easyconfigs` -* using ``eb --from-pr`` (see :ref:`github_from_pr`) +* [by providing a single easyconfig file](#by-providing-a-single-easyconfig-file) +* [via command line options](#via-command-line-options) +* [by providing a set of easyconfig files](#by-providing-a-set-of-easyconfig-files) +* [using `eb --from-pr`](integration-with-github.md#using-easyconfigs-from-pull-requests) Whenever EasyBuild searches for *explicitly specified* easyconfig files, it considers a couple of locations, i.e. (in order of preference): -(i) the local working directory -(ii) the robot search path (see :ref:`robot_search_path`) -(iii) the path to easyconfig files that are part of the active EasyBuild installation - (which is included in the default robot search path) +* (i) the local working directory +* (ii) the robot search path (see [robot search path](#searching-for-easyconfigs-the-robot-search-path)) +* (iii) the path to easyconfig files that are part of the active EasyBuild installation + (which is included in the default robot search path) These locations are only considered for easyconfig files that are specified only by filename or using a relative path, -*not* for easyconfig files that are specified via an absolute path. The dependencies are resolved using the robot search path (see :ref:`robot_search_path`). +*not* for easyconfig files that are specified via an absolute path. The dependencies are resolved using the +robot search path (see [robot search path](#searching-for-easyconfigs-the-robot-search-path)). + +!!! note -.. note:: - For easyconfig files specified on the ``eb`` command line, the *full* robot search path is only considered since - EasyBuild v2.0.0. Earlier versions only considered the local working directory and the easyconfig files that are - part of the active EasyBuild installation for *explicitly specified* easyconfig files. + For easyconfig files specified on the `eb` command line, the *full* robot search path is only considered since + EasyBuild v2.0.0. Earlier versions only considered the local working directory and the easyconfig files that are + part of the active EasyBuild installation for *explicitly specified* easyconfig files. -.. _specifying_easyconfigs_single: -By providing a single easyconfig file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### By providing a single easyconfig file -The most basic usage is to simply provide the name of an easyconfig file to the ``eb`` command. +The most basic usage is to simply provide the name of an easyconfig file to the `eb` command. EasyBuild will (try and) locate the easyconfig file, and perform the installation as specified by that easyconfig file. -For example, to build and install ``HPL`` using the ``goolf`` toolchain:: - - $ eb HPL-2.0-goolf-1.4.10.eb --robot - [...] - == building and installing GCC/4.7.2... - [...] - == building and installing goolf/1.4.10... - [...] - == building and installing HPL/2.0-goolf-1.4.10... - == fetching files... - == creating build dir, resetting environment... - == unpacking... - == patching... - == preparing... - == configuring... - == building... - == testing... - == installing... - == taking care of extensions... - == packaging... - == postprocessing... - == sanity checking... - == cleaning up... - == creating module... - == COMPLETED: Installation ended successfully - == Results of the build can be found in the log file /home/example/.local/easybuild/software/HPL/2.0-goolf-1.4.10/easybuild/easybuild-HPL-2.0-20141031.223237.log - == Build succeeded for 9 out of 9 - == temporary log file /tmp/easybuild-UOEWix/easybuild-NiswcV.log has been removed. - == temporary directory /tmp/easybuild-UOEWix has been removed. - -Then, we can actually load the freshly installed HPL module:: - - $ module load HPL/2.0-goolf-1.4.10 - $ which xhpl - /home/example/.local/easybuild/software/HPL/2.0-goolf-1.4.10/bin/xhpl - -All easyconfig file names' suffixes are ``.eb`` and follow format:: - - ``---`` - -This is a crucial design aspect, since the dependency resolution mechanism (see :ref:`use_robot`) relies upon this convention. - -.. tip:: You may wish to modify the installation prefix (e.g., using ``--prefix`` or by defining ``$EASYBUILD_PREFIX``), - in order to redefine the build/install/source path prefix to be used; default value is: ``$HOME/.local/easybuild``. - - -.. _specifying_easyconfigs_command_line: - -Via command line options -~~~~~~~~~~~~~~~~~~~~~~~~ +For example, to build and install `HPL` using the `foss/2021b` toolchain: + +```console +$ eb HPL-2.3-foss-2021b.eb --robot +[...] +== building and installing GCC/11.2.0... +[...] +== building and installing foss/2021b... +[...] +== building and installing HPL/2.3-foss-2021b... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== building... +== testing... +== installing... +== taking care of extensions... +== packaging... +== postprocessing... +== sanity checking... +== cleaning up... +== creating module... +== COMPLETED: Installation ended successfully +== Results of the build can be found in the log file /home/example/.local/easybuild/software/HPL/2.3-foss-2021b/easybuild/easybuild-HPL-2.3-20221027.223237.log +== Build succeeded for 9 out of 9 +``` -An alternative approach is to only use command line options to specify which software to build. -Refer to the ``Software search and build options`` section in the ``eb --help`` output for an overview -of the available command line options for this purpose (cfr. :ref:`basic_usage_help`). +Then, we can actually load the freshly installed HPL module: + +```console +$ module load HPL/2.3-foss-2021b +$ which xhpl +/home/example/.local/easybuild/software/HPL/2.3-foss-2021b/bin/xhpl +``` -Here is how to build and install last version of HPCG (that EasyBuild is aware of) -using the ``goolf/1.4.10`` toolchain:: +All easyconfig file names' suffixes are `.eb` and follow format: +`--` - $ eb --software-name=HPCG --toolchain=goolf,1.4.10 - [...] - == building and installing HPCG/2.1-goolf-1.4.10... - [...] - == COMPLETED: Installation ended successfully - [...] +This is a crucial design aspect, since the dependency resolution mechanism (see [robot]()) relies upon this convention. -At this point, a module ``HPCG/2.1-goolf-1.4.10`` should have been installed. +!!! tip + You may wish to modify the installation prefix (e.g., using `--prefix` or by defining `$EASYBUILD_PREFIX`), + in order to redefine the build/install/source path prefix to be used; default value is: `$HOME/.local/easybuild`; + see [Configuration](configuration.md) -.. _specifying_easyconfigs_set_of_easyconfigs: -By providing a set of easyconfig files -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Via command line options -Multiple easyconfig files can be provided as well, either directly or by specifying a directory that contains easyconfig files. +An alternative approach is to only use command line options to specify which software to build. +Refer to the `Software search and build options` section in the `eb --help` output for an overview +of the available command line options for this purpose (see also [`--help`](#command-line-help)). + +Here is how to build and install last version of HPL (that EasyBuild is aware of) +using the `foss/2021b` toolchain: + +```console +$ eb --software-name=HPL --toolchain=foss,2021b +[...] +== building and installing HPL/2.3-foss-2021b... +[...] +== COMPLETED: Installation ended successfully +[...] +``` + +At this point, a module `HPL/2.3-foss-2021b` should have been installed. -For example, to build and install both HPCG and GCC with a single command, simply list the easyconfigs for both on the -``eb`` command line (note that HPCG is not being reinstalled, since a matching module is already available):: - $ eb HPCG-2.1-goolf-1.4.10.eb GCC-4.8.3.eb - [...] - == HPCG/2.1-goolf-1.4.10 is already installed (module found), skipping - [...] - == building and installing GCC/4.8.3... - [...] - == Build succeeded for 1 out of 1 - [...] +### By providing a set of easyconfig files +Multiple easyconfig files can be provided as well, either directly or by specifying a directory that contains easyconfig files. + +For example, to build and install both HPL and GCC with a single command, simply list the easyconfigs for both on the +`eb` command line (note that HPL is not being reinstalled, since a matching module is already available): + +```console +$ eb HPL-2.3-foss-2021b.eb GCC-12.2.0.eb +[...] +== HPL/2.3-foss-2021b is already installed (module found), skipping +[...] +== building and installing GCC/12.2.0... +[...] +== Build succeeded for 1 out of 1 +[...] +``` When one or more directories are provided, EasyBuild will (recursively) traverse them to find easyconfig files. For example: -:: +```console +$ find set_of_easyconfigs/ -type f +set_of_easyconfigs/GCC-12.2.0.eb +set_of_easyconfigs/foo.txt +set_of_easyconfigs/tools/HPL-2.3-foss-2021b.eb +``` + +```console +$ eb set_of_easyconfigs/ +== temporary log file in case of crash /tmp/easybuild-1yxCvv/easybuild-NeNmZr.log +== HPL/2.3-foss-2021b is already installed (module found), skipping +== GCC/12.2.0 is already installed (module found), skipping +== No easyconfigs left to be built. +== Build succeeded for 0 out of 0 +``` + +!!! note + + EasyBuild will only pick up the files which end with `.eb` -- anything else will be ignored. - $ find set_of_easyconfigs/ -type f - set_of_easyconfigs/GCC-4.8.3.eb - set_of_easyconfigs/foo.txt - set_of_easyconfigs/tools/HPCG-2.1-goolf-1.4.10.eb +!!! tip -:: + Calling EasyBuild is designed as an *idempotent* operation; + if a module is available that matches with an provided easyconfig file, the installation will simply be skipped. - $ eb set_of_easyconfigs/ - == temporary log file in case of crash /tmp/easybuild-1yxCvv/easybuild-NeNmZr.log - == HPCG/2.1-goolf-1.4.10 is already installed (module found), skipping - == GCC/4.8.3 is already installed (module found), skipping - == No easyconfigs left to be built. - == Build succeeded for 0 out of 0 - == temporary log file /tmp/easybuild-1yxCvv/easybuild-NeNmZr.log has been removed. - == temporary directory /tmp/easybuild-1yxCvv has been removed. -.. note:: EasyBuild will only pick up the files which end with ``.eb`` ; anything else will be ignored. +## Commonly used command line options -.. tip:: Calling EasyBuild is designed as an `idempotent` operation; - if a module is available that matches with an provided easyconfig file, the installation will simply be skipped. +### Command line help +*(`--help`, `-H`)* -Commonly used command line options ----------------------------------- +Detailed information about the usage of the eb command is available via the `--help`, `-H`, `-h` help options. -Command line help, ``--help`` / ``-H`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +!!! note -Detailed information about the usage of the eb command is available via the ``--help``, ``-H``, ``-h`` help options. + `--help` / `-H` spit out the long help info (i.e. including long option names), `-h` only includes short option names. -Refer to page :ref:`basic_usage_help` for more detailed information. +!!! tip -.. note:: ``--help`` / ``-H`` spit out the long help info (i.e. including long option names), ``-h`` only includes short option names. -.. tip:: This is the best way to query for certain information, esp. recent features, since this is in sync with the actual EasyBuild version being used. + This is the best way to query for certain information, esp. recent features, since this is in sync with the actual EasyBuild version being used. -Report version, ``--version`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Report version -You can query which EasyBuild version you are using with ``--version``:: +*(`--version`)* - $ eb --version - This is EasyBuild 1.15.2 (framework: 1.15.2, easyblocks: 1.15.2) on host example.local. +You can query which EasyBuild version you are using with `--version`: + +```console +$ eb --version +This is EasyBuild 4.6.2 (framework: 4.6.2, easyblocks: 4.6.2) on host example.local. +``` -.. tip:: Asking EasyBuild to print own its version is a quick way to ensure that ``$PYTHONPATH`` - is set up correctly, so that the entire EasyBuild installation (framework and easyblocks) is available. +!!! tip -.. _list_toolchains: + Asking EasyBuild to print own its version is a quick way to ensure that `$PYTHONPATH` + is set up correctly, so that the entire EasyBuild installation (framework and easyblocks) is available. -List of known toolchains, ``--list-toolchains`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### List of known toolchains -For an overview of known toolchains, use ``eb --list-toolchains``. +*(`--list-toolchains`)* -Toolchains have brief mnemonic names, for example: +For an overview of known [toolchains](terminology.md#toolchains), use `eb --list-toolchains`. -* ``goolf`` stands for ``GCC, OpenMPI, OpenBLAS/LAPACK, FFTW and ScaLAPACK`` -* ``iimpi`` stands for ``icc/ifort, impi`` -* ``cgmvolf`` stands for ``Clang/GCC, MVAPICH2, OpenBLAS/LAPACK, FFTW`` +The complete table of available toolchains is available [here](version-specific/eb-list-toolchains.md). -The complete table of available toolchains is available at :ref:`toolchains_table`. -.. _list_easyblocks: +### List of available easyblocks -List of available easyblocks, ``--list-easyblocks`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +*(`--list-easyblocks`)* -You can obtain a list of available :ref:`easyblocks` via ``--list-easyblocks``. +You can obtain a list of available [easyblocks](terminology.md#easyblocks) via `--list-easyblocks`. -The ``--list-easyblocks`` command line option prints the easyblocks in a hierarchical way, -showing the inheritance patterns, with the "base" easyblock class ``EasyBlock`` on top. +The `--list-easyblocks` command line option prints the easyblocks in a hierarchical way, +showing the inheritance patterns, with the "base" easyblock class `EasyBlock` on top. -Software-specific easyblocks have a name that starts with ``EB_``; the ones that do not are generic easyblocks. -(cfr. :ref:`easyblocks` for the distinction between both types of easyblocks). +Software-specific easyblocks have a name that starts with `EB_`; the ones that do not are generic easyblocks. +(see [easyblocks](terminology.md#easyblocks) for the distinction between both types of easyblocks). -For example, a list of easyblocks can be obtained with:: +For example, a list of easyblocks can be obtained with: - $ eb --list-easyblocks +```shell +eb --list-easyblocks +``` To see more details about the available easyblocks, i.e., in which Python module the class is defined, and where it is -located, use ``--list-easyblocks=detailed``. +located, use `--list-easyblocks=detailed`. -Refer to page :ref:`basic_usage_easyblocks` for more information. -.. _avail_easyconfig_params: +## All available easyconfig parameters -All available easyconfig parameters, ``--avail-easyconfig-params`` / ``-a`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +*(`--avail-easyconfig-params` / `-a`)* EasyBuild provides a significant amount of easyconfig parameters. An overview of all available easyconfig parameters can be obtained via -``eb --avail-easyconfig-params``, or ``eb -a`` for short. +`eb --avail-easyconfig-params`, or `eb -a` for short. + +See [available easyconfig parameters](version-specific/easyconfig-parameters.md) for more information, +the supported parameters are a very rich set. + +Combine -a with `--easyblock`/`-e` to include parameters that are specific to a particular easyblock. For example: + +```shell +eb -a -e EB_WRF +``` + +If you want to see the full output of running this command, see [here](version-specific/eb-a-wrf.md). + + +### Enable debug logging + +*(`--debug` / `-d`)* + +Use `eb --debug`/`-d` to enable debug logging, to include all details of how EasyBuild performed a build in the log file: + +```shell +eb HPL-2.3-foss-2021b.eb --debug +``` + +!!! tip + + You may enable this by default via adding `debug = True` in your EasyBuild configuration file. -Refer to page :ref:`easyconfig_params` for more information, the possible parameters are a very rich set. +!!! note -Combine -a with ``--easyblock/-e`` to include parameters that are specific to a particular easyblock. For example:: + Debug log files are significantly larger than non-debug logs, beware! - $ eb -a -e EB_WRF -If you want to see the full output of running this command, look at :doc:`eb_a_e_EB_WRF`. +### Rebuild installation -Enable debug logging, ``--debug`` / ``-d`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +*(`--rebuild`)* -Use ``eb --debug/-d`` to enable debug logging, to include all details of how EasyBuild performed a build in the log file:: +Use `eb --rebuild` to rebuild a given easyconfig/module. - $ eb HPCG-2.1-goolf-1.4.10.eb -d +!!! warning -.. tip:: You may enable this by default via adding ``debug = True`` in your EasyBuild configuration file + Use with care, since the reinstallation of existing modules will be done without requesting confirmation first! -.. note:: Debug log files are significantly larger than non-debug logs, so be aware. +!!! tip -.. _rebuild_option: + Combine `--rebuild` with `--dry-run` to get a good view on which installations will be rebuilt + (see also [Getting an overview of planned installations](#getting-an-overview-of-planned-installations)). -Rebuild installation, ``--rebuild`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +!!! note -Use ``eb --rebuild`` to rebuild a given easyconfig/module. + To create a backup of existing modules that are regenerated when `--rebuild` is used, + use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). -.. warning:: Use with care, since the reinstallation of existing modules will be done without requesting confirmation first! -.. tip:: Combine ``--rebuild`` with ``--dry-run`` to get a good view on which installations will be rebuilt. - (cfr. :ref:`get_an_overview`) +### Forced reinstallation -.. note:: To create a backup of existing modules that are regenerated when ``--rebuild`` is used, - use ``--backup-modules`` (see also :ref:`backup_modules`). +*(`--force` / `-f`)* -.. _force_option: +Use `eb --force`/`-f` to force the reinstallation of a given easyconfig/module. +The behavior of `--force` is the same as `--rebuild` and `--ignore-osdeps`. -Forced reinstallation, ``--force`` / ``-f`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +!!! warning -Use ``eb --force``/``-f`` to force the reinstallation of a given easyconfig/module. -The behavior of ``--force`` is the same as ``--rebuild`` and ``--ignore-osdeps``. + Use with care, since the reinstallation of existing modules will be done without requesting confirmation first! -.. warning:: Use with care, since the reinstallation of existing modules will be done without requesting confirmation first! +!!! tip -.. tip:: Combine ``--force`` with ``--dry-run`` to get a good view on which installations will be forced. - (cfr. :ref:`get_an_overview`) + Combine `--force` with `--dry-run` to get a good view on which installations will be forced + (see also [Getting an overview of planned installations](#getting-an-overview-of-planned-installations)). -.. note:: To create a backup of existing modules that are regenerated when ``--force`` is used, - use ``--backup-modules`` (see also :ref:`backup_modules`). +!!! note -.. _searching_for_easyconfigs: + To create a backup of existing modules that are regenerated when `--rebuild` is used, + use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). -Searching for easyconfigs, ``--search`` / ``-S`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Searching for available easyconfig files can be done using the ``--search`` (long output) and ``-S`` (short output) +### Searching for easyconfigs + +*(`--search` / `-S`)* + +Searching for available easyconfig files can be done using the `--search` (long output) and `-S` (short output) command line options. All easyconfig files available in the robot search path are considered -(see :ref:`robot_search_path`), and searching is done `case-insensitive`. - -For example, to see which easyconfig files are available for the software package named `Mesquite`:: - - $ eb --search mesquite - == temporary log file in case of crash /tmp/eb-_TYdTf/easybuild-iRJ2vb.log - == Searching (case-insensitive) for 'mesquite' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs - * /Users/kehoste/work/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite/Mesquite-2.3.0-goolf-1.4.10.eb - * /Users/kehoste/work/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite/Mesquite-2.3.0-ictce-4.1.13.eb - * /Users/kehoste/work/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite/Mesquite-2.3.0-ictce-5.3.0.eb - == temporary log file(s) /tmp/eb-_TYdTf/easybuild-iRJ2vb.log* have been removed. - == temporary directory /tmp/eb-_TYdTf has been removed. - -The same query with ``-S`` is more readable, when there is a joint path that can be collapsed to a variable like ``$CFGS1``:: - - $ eb -S mesquite - == temporary log file in case of crash /tmp/eb-5diJjn/easybuild-nUXlkj.log - == Searching (case-insensitive) for 'mesquite' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs - CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite - * $CFGS1/Mesquite-2.3.0-goolf-1.4.10.eb - * $CFGS1/Mesquite-2.3.0-ictce-4.1.13.eb - * $CFGS1/Mesquite-2.3.0-ictce-5.3.0.eb - == temporary log file(s) /tmp/eb-5diJjn/easybuild-nUXlkj.log* have been removed. - == temporary directory /tmp/eb-5diJjn has been removed. +(see [robot search path](#searching-for-easyconfigs-the-robot-search-path)), and searching is done *case-insensitive*. + +For example, to see which easyconfig files are available for the software package named `Mesquite`: + +```console +$ eb --search mesquite +== temporary log file in case of crash /tmp/eb-_TYdTf/easybuild-iRJ2vb.log +== Searching (case-insensitive) for 'mesquite' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs +* /home/example/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite/Mesquite-2.3.0-GCCcore-8.3.0.eb +``` + +The same query with `-S` is more readable, when there is a joint path that can be collapsed to a variable like `$CFGS1`: + +```console +$ eb -S mesquite +== temporary log file in case of crash /tmp/eb-5diJjn/easybuild-nUXlkj.log +== Searching (case-insensitive) for 'mesquite' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs +CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesquite +* $CFGS1/Mesquite-2.3.0-GCCcore-8.3.0.eb +``` For more specific searching, a regular expression pattern can be supplied (since EasyBuild v2.1.1). -For example, to search which easyconfig files are available for GCC v4.6.x, without listing easyconfig files that use -GCC v4.6.x as a toolchain:: - - $ eb -S ^GCC-4.6 - == temporary log file in case of crash /tmp/eb-PpwTwm/easybuild-b8yrOG.log - == Searching (case-insensitive) for '^GCC-4.6' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs - CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/g/GCC - * $CFGS1/GCC-4.6.3-CLooG-PPL.eb - * $CFGS1/GCC-4.6.3.eb - * $CFGS1/GCC-4.6.4.eb - == temporary log file(s) /tmp/eb-PpwTwm/easybuild-b8yrOG.log* have been removed. - == temporary directory /tmp/eb-PpwTwm has been removed. - -Or, to find all easyconfig files for Python versions 2.7.8 and 2.7.9 that use the ``intel`` toolchain:: - - $ eb -S '^Python-2.7.[89].*intel' - == temporary log file in case of crash /tmp/eb-Dv5LEJ/easybuild-xpGGSF.log - == Searching (case-insensitive) for '^Python-2.7.[89].*intel' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs - CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/p/Python - * $CFGS1/Python-2.7.8-intel-2014.06.eb - * $CFGS1/Python-2.7.8-intel-2014b.eb - * $CFGS1/Python-2.7.8-intel-2015a.eb - * $CFGS1/Python-2.7.9-intel-2015a-bare.eb - * $CFGS1/Python-2.7.9-intel-2015a.eb - == temporary log file(s) /tmp/eb-Dv5LEJ/easybuild-xpGGSF.log* have been removed. - == temporary directory /tmp/eb-Dv5LEJ has been removed. - -.. note:: Prior to EasyBuild v2.1.1, the full path to easyconfig files was considered when matching the search pattern. - Starting with EasyBuild v2.1.1, only the filename of the easyconfig file itself is taken into account. - -.. _use_robot: - -Enabling dependency resolution, ``--robot`` / ``-r`` and ``--robot-paths`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +For example, to search which easyconfig files are available for GCC v11.x, without listing easyconfig files that use +GCC v11.x as a toolchain: + +```console +$ eb -S ^GCC-11. +== temporary log file in case of crash /tmp/eb-PpwTwm/easybuild-b8yrOG.log +== Searching (case-insensitive) for '^GCC-11.' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs +CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/g/GCC + * $CFGS1/GCC-11.1.0.eb + * $CFGS1/GCC-11.2.0.eb + * $CFGS1/GCC-11.3.0.eb +``` + +Or, to find all easyconfig files for matplotlib versions 3.5.1 and 3.5.2 that use the `intel` toolchain: + +```console +$ eb -S '^matplotlib-3.5.[12].*intel' +== temporary log file in case of crash /tmp/eb-Dv5LEJ/easybuild-xpGGSF.log +== Searching (case-insensitive) for '^matplotlib-3.5.[12].*intel' in /home/example/easybuild-easyconfigs/easybuild/easyconfigs +CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/m/matplotlib + * $CFGS1/matplotlib-3.5.1-intel-2020b.eb + * $CFGS1/matplotlib-3.5.2-intel-2022a.eb +``` + + +## Enabling dependency resolution + +*(`--robot` / `-r` and `--robot-paths`)* EasyBuild supports installing an entire software stack, including the required toolchain -if needed, with a single ``eb`` invocation. - -To enable dependency resolution, use the ``--robot`` command line option (or ``-r`` for short):: - - $ eb mpiBLAST-1.6.0-goolf-1.4.10.eb --robot - [...] - == building and installing GCC/4.7.2... - [...] - == building and installing hwloc/1.6.2-GCC-4.7.2... - [...] - == building and installing OpenMPI/1.6.4-GCC-4.7.2... - [...] - == building and installing gompi/1.4.10... - [...] - == building and installing OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2... - [...] - == building and installing FFTW/3.3.3-gompi-1.4.10... - [...] - == building and installing ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2... - [...] - == building and installing goolf/1.4.10... - [...] - == building and installing mpiBLAST/1.6.0-goolf-1.4.10... - [...] - == Build succeeded for 9 out of 9 +if needed, with a single `eb` invocation. + +To enable dependency resolution, use the `--robot` command line option (or `-r` for short): + +```console +$ eb binutils-2.39.eb --robot +[...] +== building and installing zlib/1.2.12... +[...] +== building and installing M4/1.4.19... +[...] +== building and installing Bison/3.8.2... +[...] +== building and installing flex/2.6.4... +[...] +== building and installing binutils/2.39... +[...] +== Build succeeded for 5 out of 5 +``` The dependency resolution mechanism will construct a full dependency graph for the software package(s) being installed, after which a list of dependencies is composed for which no module is available yet. Each of the retained dependencies will then be built and installed, in the required order as indicated by the dependency graph. -.. tip:: Using ``--robot`` is particularly useful for software packages that have an extensive list of dependencies, - or when reinstalling software using a different compiler toolchain - (you can use the ``--try-toolchain`` command line option in combination with ``--robot``). +!!! tip + + Using `--robot` is particularly useful for software packages that have an extensive list of dependencies, + or when reinstalling software using a different compiler toolchain + (you can use the `--try-toolchain` command line option in combination with `--robot`). -.. note:: Unless dependency resolution is enabled, EasyBuid requires that modules are available for every dependency. - If ``--robot`` is not used and one or more modules are missing, ``eb`` will exit with an error stating that a module - for a particular dependency could not be found. For example:: +!!! warning - add_dependencies: no module 'GCC/4.7.2' found for dependency {...} + Unless dependency resolution is enabled, EasyBuid requires that modules are available for every dependency. + If `--robot` is not used and one or more modules are missing, `eb` will exit with an error stating that a module + for a particular dependency could not be found. For example: -.. _robot_search_path: + ```console + Missing modules for dependencies (use --robot?): zlib/1.2.12 + ``` -Searching for easyconfigs: the robot search path -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Searching for easyconfigs: the robot search path For each dependency that does not have a matching module installed yet, EasyBuild requires a corresponding easyconfig -file. If no such easyconfig file was specified on the ``eb`` command line, the dependency resolution mechanism will try +file. If no such easyconfig file was specified on the `eb` command line, the dependency resolution mechanism will try to locate one in the `robot search path`. -Searching for easyconfigs is done based on filename (see also :ref:`what_is_an_easyconfig`), with filenames being derived -from the dependency specification (i.e. software name/version, toolchain and version suffix). For each entry in the robot -search path, a couple of different filepaths are considered, mostly determined by the software name. +Searching for easyconfigs is done based on filename (see also [What is an easyconfig (file)?](#what-is-an-easyconfig-file)), +with filenames being derived from the dependency specification (i.e. software name/version, toolchain and version suffix). +For each entry in the robot search path, a couple of different filepaths are considered, mostly determined by the software name. + +For example, when looking for an easyconfig for `OpenMPI` version `4.1.4` and version suffix `-test` with toolchain +`GCC/11.3.0`, the following filepaths are considered (relative to each entry in the robot search path): + +* `OpenMPI/4.1.4-GCC-11.3.0-test.eb` +* `OpenMPI/OpenMPI-4.1.4-GCC-11.3.0-test.eb` +* `o/OpenMPI/OpenMPI-4.1.4-GCC-11.3.0-test.eb` +* `OpenMPI-4.1.4-GCC-11.3.0-test.eb` -For example, when looking for an easyconfig for ``OpenMPI`` version ``1.6.4`` and version suffix ``-test`` with toolchain -``GCC/4.7.2``, the following filepaths are considered (relative to each entry in the robot search path): +!!! note -* ``OpenMPI/1.6.4-GCC-4.7.2-test.eb`` -* ``OpenMPI/OpenMPI-1.6.4-GCC-4.7.2-test.eb`` -* ``o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2-test.eb`` -* ``OpenMPI-1.6.4-GCC-4.7.2-test.eb`` + Sometimes easyconfig files are needed even when the modules for the dependencies are already available, + i.e., whenever the information provided by the dependency specification (software name/version, toolchain and version + suffix) is not sufficient. This is the case when using `--dry-run` to construct the complete dependency graph, or + when the active module naming scheme requires some additional information (e.g., the `moduleclass`). -.. note:: Sometimes easyconfig files are needed even when the modules for the dependencies are already available, - i.e., whenever the information provided by the dependency specification (software name/version, toolchain and version - suffix) is not sufficient. This is the case when using ``--dry-run`` to construct the complete dependency graph, or - when the active module naming scheme requires some additional information (e.g., the ``moduleclass``). +!!! warning -.. note:: If EasyBuild is unable to locate required easyconfigs, an appropriate error message will be shown. For example:: + If EasyBuild is unable to locate required easyconfigs, an appropriate error message will be shown. For example: + ```console Irresolvable dependencies encountered: GCC/4.7.2 + ``` - or:: + or: + ```console Failed to find easyconfig file 'GCC-4.7.2.eb' when determining module name for: {...} + ``` -Default robot search path -+++++++++++++++++++++++++ +### Default robot search path By default, EasyBuild will only include the collection of easyconfig files that is part of the EasyBuild installation in the robot search path. More specifically, only directories listed in the `Python search path` (partially specified -by the ``$PYTHONPATH`` environment variable) that contain a subdirectory named ``easybuild/easyconfigs`` are considered +by the `$PYTHONPATH` environment variable) that contain a subdirectory named `easybuild/easyconfigs` are considered part of the robot search path (in the order they are encountered). -.. _controlling_robot_search_path: -Controlling the robot search path -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Controlling the robot search path -To control the robot search path, you can specify a (colon-separated list of) path(s) to ``--robot``/``-r`` and/or -``--robot-paths`` (or, equivalently, ``$EASYBUILD_ROBOT`` and/or ``$EASYBUILD_ROBOT_PATHS``):: +To control the robot search path, you can specify a (colon-separated list of) path(s) to `--robot`/`-r` and/or +`--robot-paths` (or, equivalently, `$EASYBUILD_ROBOT` and/or `$EASYBUILD_ROBOT_PATHS`): - eb --robot=$PWD:$HOME ... +```shell +eb --robot=$PWD:$HOME ... +``` These two configuration options each serve a particular purpose, and together define the robot search path: -* ``--robot``, ``-r``: - - * intended to be used (only) as a command line option to ``eb`` (although it can also be defined through another - configuration type) - * enables the dependency resolution mechanism (disabled by default) - * optionally a list of paths can be specified, which is included *first* in the robot search path - * by default, the corresponding list of paths is *empty* - -* ``--robot-paths``: - - * intended to be defined in an EasyBuild configuration file, or via ``$EASYBUILD_ROBOT_PATHS`` - * does *not* enable the dependency resolution mechanism - * the specified list of paths is included *last* in the robot search path - * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed - * **note**: setting this configuration option implies redefining the default robot search path, unless a - prepending/appending list of paths is specified, see :ref:`robot_search_path_prepend_append` - -For both options, the list of paths should be specified as a colon-separated (``:``) list. - -By combining ``--robot`` and ``--robot-paths`` using the different configuration types (see also -:ref:`configuration_types`), you have full control over the robot search path: which paths are included, +* `--robot`, `-r`: + * intended to be used (only) as a command line option to `eb` (although it can also be defined through another + configuration type); + * enables the dependency resolution mechanism (which is disabled by default); + * optionally a list of paths can be specified, which is included *first* in the robot search path; + * by default, the corresponding list of paths is *empty*; +* `--robot-paths`: + * intended to be defined in an EasyBuild configuration file, or via `$EASYBUILD_ROBOT_PATHS`; + * does *not* enable the dependency resolution mechanism; + * the specified list of paths is included *last* in the robot search path; + * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed; + * **note**: setting this configuration option implies redefining the default robot search path, unless a + prepending/appending list of paths is specified, see + [Prepending/appending to the default robot search path](#prependingappending-to-the-default-robot-search-path); + +For both options, the list of paths should be specified as a colon-separated (`:`) list. + +By combining `--robot` and `--robot-paths` using the different configuration types (see also +[Supported configuration types](#supported-configuration-types)), you have full control over the robot search path: which paths are included, the order of those paths, whether or not the easyconfig files that are part of the EasyBuild installation should be considered, etc. -A constant named ``DEFAULT_ROBOT_PATHS`` is available that can be used (only) in EasyBuild configuration files to refer +A constant named `DEFAULT_ROBOT_PATHS` is available that can be used (only) in EasyBuild configuration files to refer to the default robot search path, i.e. the path to the easyconfigs that are part of the EasyBuild installation. For more information on using constants in EasyBuild configuration files, see -:ref:`configuration_file_templates_constants`. +[Templates and constants supported in configuration files](#templates-and-constants-supported-in-configuration-files). + +!!! tip + + Only use `--robot` to enable the dependency resolution mechanism; define `robot-paths` in your EasyBuild + configuration file or via `$EASYBUILD_ROBOT_PATHS` to specify which sets of easyconfig files EasyBuild + should consider, and in which order. By means of exception, a path can be specified to `--robot` to give a specific + set of easyconfig files precedence over others, for example when testing modified easyconfig files. -.. tip:: - Only use ``--robot`` to enable the dependency resolution mechanism; define ``robot-paths`` in your EasyBuild - configuration file or via ``$EASYBUILD_ROBOT_PATHS`` to specify which sets of easyconfig files EasyBuild - should consider, and in which order. By means of exception, a path can be specified to ``--robot`` to give a specific - set of easyconfig files precedence over others, for example when testing modified easyconfig files. +!!! note -.. note:: The paths specified on the configuration type with the highest order of preference *replace* any paths - specified otherwise, i.e. values are not retained across configuration types. That is: ``--robot`` *overrides* - the value in ``$EASYBUILD_ROBOT``, ``$EASYBUILD_ROBOT_PATHS`` *overrides* the ``robot-paths`` specification in an - EasyBuild configuration file, etc. Of course, the value specified for ``--robot`` does not directly affect the value - specified for ``--robot-paths``, on any configuration level, and vice versa. For more information on the relation - between the different configuration types, see :ref:`configuration_types`. + The paths specified on the configuration type with the highest order of preference *replace* any paths + specified otherwise, i.e. values are not retained across configuration types. That is: `--robot` *overrides* + the value in `$EASYBUILD_ROBOT`, `$EASYBUILD_ROBOT_PATHS` *overrides* the `robot-paths` specification in an + EasyBuild configuration file, etc. Of course, the value specified for `--robot` does not directly affect the value + specified for `--robot-paths`, on any configuration level, and vice versa. For more information on the relation + between the different configuration types, see [Supported configuration types](#supported-configuration-types). -.. _robot_search_path_prepend_append: -Prepending and/or appending to the default robot search path -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +### Prepending/appending to the default robot search path -Prepending or appending to the default robot search path is supported via the ``--robot-paths`` configuration option. +Prepending or appending to the default robot search path is supported via the `--robot-paths` configuration option. -To *prepend* one or more paths, a list of paths followed by a '``:``' should be specified. +To *prepend* one or more paths, a list of paths followed by a '`:`' should be specified. -Analogously, to *append* one or more paths, a list of paths preceded by a '``:``' should be specified. +Analogously, to *append* one or more paths, a list of paths preceded by a '`:`' should be specified. For example: -* ``export EASYBUILD_ROBOT_PATHS=/tmp/$USER:`` specifies to prepend ``/tmp/$USER`` to the robot search path -* ``--robot-paths :$HOME/eb:$HOME/test`` specifies to append ``$HOME/eb`` and ``$HOME/test`` to the robot search path (in that order) -* ``--robot-paths=/tmp/$USER::$HOME/test`` specifies to prepend ``/tmp/$USER`` *and* append ``$HOME/test`` to the robot search path +* "`export EASYBUILD_ROBOT_PATHS=/tmp:`" specifies to *prepend* `/tmp` to the robot search path; +* "`--robot-paths :/tmp:$HOME/test`" specifies to *append* `/tmp` and `$HOME/test` to the robot search path (in that order); +* "`--robot-paths=/tmp::$HOME/test`" specifies to *prepend* `/tmp` *and* to *append* `$HOME/test` to the robot search path; -Example use case -++++++++++++++++ +### Example use case For example, say we want to configure EasyBuild to behave as follows w.r.t. the robot search path: -* (always) prefer easyconfig files in the archive/repository over the ones that are included in the EasyBuild installation (i) -* consider easyconfig files located in the current directory or home directory first (in that order), before any others (ii) +* (i) (always) prefer easyconfig files in the archive/repository over the ones that are included in the EasyBuild installation; +* (ii) consider easyconfig files located in the current directory or home directory first (in that order), before any others; Matching setup: -* satisfy (i) using ``robot-paths`` in one of the active EasyBuild configuration files (see also - :ref:`list_of_configuration_files`):: +* satisfy (i) using `robot-paths` in one of the active EasyBuild configuration files (see also + [List of configuration files](#list-of-used-configuration-files)): - [basic] - repositorypath = /home/example/easybuild/easyconfigs_archive - robot-paths = %(repositorypath)s:%(DEFAULT_ROBOT_PATHS)s + ```ini + [basic] + repositorypath = /home/example/easybuild/easyconfigs_archive + robot-paths = %(repositorypath)s:%(DEFAULT_ROBOT_PATHS)s + ``` -* satisfy (ii) via ``--robot`` on the ``eb`` command line:: +* satisfy (ii) via `--robot` on the `eb` command line: - eb mpiBLAST-1.6.0-goolf-1.4.10.eb --robot $PWD:$HOME + ```shell + eb example.eb --robot $PWD:$HOME + ``` -.. _get_an_overview: +## Getting an overview of planned installations -Getting an overview of planned installations ``--dry-run`` / ``-D`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +*(`--dry-run` / `-D`)* -You can do a "dry-run" overview by supplying ``-D/--dry-run`` (typically combined with ``--robot``, in the form of ``-Dr``):: +You can do a "dry-run" overview by supplying `-D/--dry-run`: - $ eb mpiBLAST-1.6.0-goolf-1.4.10.eb -Dr - == temporary log file in case of crash /tmp/easybuild-vyNQhw/easybuild-pO8EJv.log - Dry run: printing build status of easyconfigs and dependencies - CFGS=/home/example/.local/easybuild/software/EasyBuild/1.15.2/lib/python2.7/site-packages/easybuild_easyconfigs-1.15.2.0-py2.7.egg/easybuild/easyconfigs - * [x] $CFGS/g/GCC/GCC-4.7.2.eb (module: GCC/4.7.2) - * [x] $CFGS/h/hwloc/hwloc-1.6.2-GCC-4.7.2.eb (module: hwloc/1.6.2-GCC-4.7.2) - * [x] $CFGS/o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2.eb (module: OpenMPI/1.6.4-GCC-4.7.2) - * [x] $CFGS/g/gompi/gompi-1.4.10.eb (module: gompi/1.4.10) - * [ ] $CFGS/o/OpenBLAS/OpenBLAS-0.2.6-gompi-1.4.10-LAPACK-3.4.2.eb (module: OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2) - * [ ] $CFGS/f/FFTW/FFTW-3.3.3-gompi-1.4.10.eb (module: FFTW/3.3.3-gompi-1.4.10) - * [ ] $CFGS/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2.eb (module: ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2) - * [ ] $CFGS/g/goolf/goolf-1.4.10.eb (module: goolf/1.4.10) - * [ ] $CFGS/m/mpiBLAST/mpiBLAST-1.6.0-goolf-1.4.10.eb (module: mpiBLAST/1.6.0-goolf-1.4.10) - == temporary log file /tmp/easybuild-vyNQhw/easybuild-pO8EJv.log has been removed. - == temporary directory /tmp/easybuild-vyNQhw has been removed. +```console +$ eb binutils-2.39.eb -D +== Temporary log file in case of crash /tmp/eb-3lh496ml/easybuild-hxmie76g.log +Dry run: printing build status of easyconfigs and dependencies +CFGS=/Volumes/work/easybuild-easyconfigs/easybuild/easyconfigs + * [x] $CFGS/z/zlib/zlib-1.2.12.eb (module: zlib/1.2.12) + * [x] $CFGS/m/M4/M4-1.4.19.eb (module: M4/1.4.19) + * [x] $CFGS/b/Bison/Bison-3.8.2.eb (module: Bison/3.8.2) + * [ ] $CFGS/f/flex/flex-2.6.4.eb (module: flex/2.6.4) + * [ ] $CFGS/b/binutils/binutils-2.39.eb (module: binutils/2.39) +``` Note how the different status symbols denote distinct handling states by EasyBuild: -* ``[ ]`` The build is not available, EasyBuild will deliver it -* ``[x]`` The build is available, EasyBuild will skip building this module -* ``[F]`` The build is available, however EasyBuild has been asked to force a rebuild via ``--force`` and will do so -* ``[R]`` The build is available, and the application will be rebuilt as request via ``--rebuild`` +* `[ ]` The build is not available, EasyBuild will deliver it +* `[x]` The build is available, EasyBuild will skip building this module +* `[F]` The build is available, however EasyBuild has been asked to force a rebuild via `--force` and will do so +* `[R]` The build is available, and the application will be rebuilt as request via `--rebuild` -.. note:: Since EasyBuild v2.4.0, a detailed overview of the build and install procedure that EasyBuild - will be execute can be obtained using ``--extended-dry-run`` or ``-x``, see :ref:`extended_dry_run`. +!!! note -.. _eb_missing: + Since EasyBuild v2.4.0, a detailed overview of the build and install procedure that EasyBuild + will be execute can be obtained using `--extended-dry-run` or `-x`, see [Extended dry run](extended-dry-run.md). -Getting an overview of missing installations ``--missing-modules`` / ``-M`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Since EasyBuild v3.9.1, you can obtain a list of missing installations (i.e. easyconfigs for which -no corresponding environment module file is available yet) using ``eb --missing-modules`` (or ``-eb -M`` for short):: +## Getting an overview of missing installations - $ eb TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb --missing-modules - == temporary log file in case of crash /tmp/eb-bjCz9b/easybuild-CSAvhK.log +*(`--missing-modules` / `-M`)* - 2 out of 54 required modules missing: +Since EasyBuild v3.9.1, you can obtain a list of missing installations (i.e. easyconfigs for which +no corresponding environment module file is available yet) using `eb --missing-modules` (or `-eb -M` for short): + +```console +$ eb TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb --missing-modules +== temporary log file in case of crash /tmp/eb-bjCz9b/easybuild-CSAvhK.log - * Bazel/0.20.0-GCCcore-8.2.0 (Bazel-0.20.0-GCCcore-8.2.0.eb) - * TensorFlow/1.13.1-foss-2019a-Python-3.7.2 (TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb) +2 out of 54 required modules missing: - == Temporary log file(s) /tmp/eb-bjCz9b/easybuild-CSAvhK.log* have been removed. - == Temporary directory /tmp/eb-bjCz9b has been removed. +* Bazel/0.20.0-GCCcore-8.2.0 (Bazel-0.20.0-GCCcore-8.2.0.eb) +* TensorFlow/1.13.1-foss-2019a-Python-3.7.2 (TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb) +``` Note that dependency resolution is enabled automatically -(i.e., ``--robot`` is implied when using ``--missing-modules`` or ``-M``). +(i.e., `--robot` is implied when using `--missing-modules` or `-M`). -.. _tweaking_easyconfigs_using_try: -Tweaking existing easyconfig files, using ``--try-*`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## Tweaking existing easyconfig files -Making minor changes to existing easyconfig files can be done straight from the ``eb`` command line. +*(`--try-*`)* + +Making minor changes to existing easyconfig files can be done straight from the `eb` command line. This way, having to manually copying and editing easyconfig files can be avoided. -Tweaking existing easyconfig files can be done using the ``--try-*`` command line options. +Tweaking existing easyconfig files can be done using the `--try-*` command line options. For each of the software build options that can be used as an alternative to specifying easyconfig file names, -a matching ``--try-X`` command line options is available: - -* ``--try-toolchain`` to try using the toolchain with the given name and version - - * format: ``--try-toolchain=,`` - * ``--try-toolchain-name`` to try using the latest toolchain version of a toolchain - * ``--try-toolchain-version`` to try using a different toolchain version - -* ``--try-software-version`` to try building a different software version -* ``--try-amend`` to try tweaking a different easyconfig parameter - - * format: ``--try-amend==`` - * only supports string and list-of-strings value types - -For example, to build and install WRF and its dependencies with a different toolchain version:: - - $ eb WRF-3.5.1-goolf-1.4.10-dmpar.eb --try-toolchain-version=1.5.14 -Dr - == temporary log file in case of crash /tmp/easybuild-Y9WApt/easybuild-VmPiOH.log - Dry run: printing build status of easyconfigs and dependencies - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-4.8.2.eb (module: GCC/4.8.2) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/h/hwloc/hwloc-1.8.1-GCC-4.8.2.eb (module: hwloc/1.8.1-GCC-4.8.2) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.2.eb (module: OpenMPI/1.6.5-GCC-4.8.2) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-1.5.14.eb (module: gompi/1.5.14) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.2.8-gompi-1.5.14-LAPACK-3.5.0.eb (module: OpenBLAS/0.2.8-gompi-1.5.14-LAPACK-3.5.0) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/f/FFTW/FFTW-3.3.4-gompi-1.5.14.eb (module: FFTW/3.3.4-gompi-1.5.14) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-1.5.14-OpenBLAS-0.2.8-LAPACK-3.5.0.eb (module: ScaLAPACK/2.0.2-gompi-1.5.14-OpenBLAS-0.2.8-LAPACK-3.5.0) - * [x] /home/example/work/easybuild-easyconfigs/easybuild/easyconfigs/g/goolf/goolf-1.5.14.eb (module: goolf/1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/zlib-1.2.7-goolf-1.5.14.eb (module: zlib/1.2.7-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/Szip-2.1-goolf-1.5.14.eb (module: Szip/2.1-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/ncurses-5.9-goolf-1.5.14.eb (module: ncurses/5.9-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/flex-2.5.37-goolf-1.5.14.eb (module: flex/2.5.37-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/M4-1.4.16-goolf-1.5.14.eb (module: M4/1.4.16-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/JasPer-1.900.1-goolf-1.5.14.eb (module: JasPer/1.900.1-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/HDF5-1.8.10-patch1-goolf-1.5.14.eb (module: HDF5/1.8.10-patch1-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/tcsh-6.18.01-goolf-1.5.14.eb (module: tcsh/6.18.01-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/Bison-2.7-goolf-1.5.14.eb (module: Bison/2.7-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/Doxygen-1.8.3.1-goolf-1.5.14.eb (module: Doxygen/1.8.3.1-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/netCDF-4.2.1.1-goolf-1.5.14.eb (module: netCDF/4.2.1.1-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/netCDF-Fortran-4.2-goolf-1.5.14.eb (module: netCDF-Fortran/4.2-goolf-1.5.14) - * [ ] /tmp/easybuild-Y9WApt/tweaked_easyconfigs/WRF-3.5.1-goolf-1.5.14-dmpar.eb (module: WRF/3.5.1-goolf-1.5.14-dmpar) - == temporary log file /tmp/easybuild-Y9WApt/easybuild-VmPiOH.log has been removed. - == temporary directory /tmp/easybuild-Y9WApt has been removed. - -.. note:: The ``--try-*`` command line options behave as expected when combined with ``--robot``. For example: a modified toolchain specified via ``--try-toolchain`` only trickles down until the toolchain level (not deeper). This makes for a particularly powerful combo for rebuilding entire software stacks using a different toolchain. - -.. note:: Modifying the software version does **not** trickle down the entire software stack, even when combined with ``--robot``, since a software version is tied to a particular software package. - +a matching `--try-X` command line options is available: + +* `--try-toolchain` to try using the toolchain with the given name and version + * format: `--try-toolchain=,` + * `--try-toolchain-name` to try using the latest toolchain version of a toolchain + * `--try-toolchain-version` to try using a different toolchain version + +* `--try-software-version` to try building a different software version + +* `--try-amend` to try tweaking a different easyconfig parameter + * format: `--try-amend==` + * only supports string and list-of-strings value types + +For example, to build and install BWA and its dependencies with a different toolchain version: + +```console +$ eb BWA-0.7.17-GCC-10.3.0.eb --try-toolchain-version 11.2.0 -Dr +== Temporary log file in case of crash /tmp/eb-93m2gyhf/easybuild-_c86e81n.log +Dry run: printing build status of easyconfigs and dependencies + ... + * [x] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-11.2.0.eb (module: GCC/11.2.0) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/g/groff/groff-1.22.4-GCCcore-11.2.0.eb (module: groff/1.22.4-GCCcore-11.2.0) + * [ ] /tmp/eb-93m2gyhf/tweaked_dep_easyconfigs/expat-2.2.9-GCCcore-11.2.0.eb (module: expat/2.2.9-GCCcore-11.2.0) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/n/ncurses/ncurses-6.2-GCCcore-11.2.0.eb (module: ncurses/6.2-GCCcore-11.2.0) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/l/libreadline/libreadline-8.1-GCCcore-11.2.0.eb (module: libreadline/8.1-GCCcore-11.2.0) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/p/pkgconf/pkgconf-1.8.0.eb (module: pkgconf/1.8.0) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/o/OpenSSL/OpenSSL-1.1.eb (module: OpenSSL/1.1) + * [ ] /home/example/easybuild-easyconfigs/easybuild/easyconfigs/d/DB/DB-18.1.40-GCCcore-11.2.0.eb (module: DB/18.1.40-GCCcore-11.2.0) + * [ ] /tmp/eb-93m2gyhf/tweaked_dep_easyconfigs/Perl-5.32.1-GCCcore-11.2.0.eb (module: Perl/5.32.1-GCCcore-11.2.0) + * [ ] /tmp/eb-93m2gyhf/tweaked_easyconfigs/BWA-0.7.17-GCC-11.2.0.eb (module: BWA/0.7.17-GCC-11.2.0) ``` + +!!! note + + The `--try-*` command line options behave as expected when combined with `--robot`. + + For example: a modified toolchain specified via `--try-toolchain` only trickles down until the toolchain level (not deeper). + This makes for a particularly powerful combo for rebuilding entire software stacks using a different toolchain. + +!!! note + + Modifying the software version does **not** trickle down the entire software stack, even when combined with `--robot`, + since a software version is tied to a particular software package. diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 8fba2ede54..bada1a8a23 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -1,5 +1,9 @@ # Writing easyconfig files + +## What is an easyconfig (file)? + + !!! warning This page will soon replace . From 44ed54ff6eb85511e2b280e6cb4721db9f042d01 Mon Sep 17 00:00:00 2001 From: James Carpenter Date: Fri, 4 Nov 2022 15:49:25 +0000 Subject: [PATCH 3/7] add new-style internal links --- docs/configuration.md | 26 ++++++++++------------- docs/terminology.md | 4 ++-- docs/using-easybuild.md | 36 ++++++++++++++++---------------- docs/writing-easyconfig-files.md | 4 ++-- 4 files changed, 33 insertions(+), 37 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 8db87c2ba8..91f6ef5187 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,12 +1,14 @@ -# Configuring EasyBuild +# Configuring EasyBuild {: #configuring_easybuild } -## Supported configuration types +## Supported configuration types {: #configuration_types } -### Configuration file(s) +### Configuration file(s) {: #configuration_file } -#### List of used configuration files +#### List of used configuration files {: #list_of_configuration_files } -#### Templates and constants supported in configuration files +#### Templates and constants supported in configuration files {: #configuration_file_templates_constants } + +### Easyconfigs repository {: #easyconfigs_repo } !!! warning This page will soon replace . @@ -38,11 +40,8 @@ A demo on configuring EasyBuild is available :ref:`here `. .. _configuration_types: -``` - -## Supported configuration types - -```rst +Supported configuration types +----------------------------- Configuring EasyBuild can be done by: @@ -534,11 +533,8 @@ so you do not need to adjust ``$MODULEPATH`` every time you start a new session. .. _easyconfigs_repo: -``` - -### Easyconfigs repository (``--repository``, ``--repositorypath``) - -```rst +Easyconfigs repository (``--repository``, ``--repositorypath``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *default*: ``FileRepository`` at ``$HOME/.local/easybuild/ebfiles_repo`` (determined via :ref:`prefix`) diff --git a/docs/terminology.md b/docs/terminology.md index 59a5c2e2ca..473577549b 100644 --- a/docs/terminology.md +++ b/docs/terminology.md @@ -1,6 +1,6 @@ -# EasyBuild terminology +# EasyBuild terminology {: #concepts_and_terminology } -## Easyconfig files +## Easyconfig files {: #easyconfig_files } !!! warning This page will soon replace . diff --git a/docs/using-easybuild.md b/docs/using-easybuild.md index 632995d624..391c425967 100644 --- a/docs/using-easybuild.md +++ b/docs/using-easybuild.md @@ -1,4 +1,4 @@ -# Using the EasyBuild command line +# Using the EasyBuild command line {: #using_the_easybuild_command_line } Basic usage of EasyBuild is described in the following sections, covering the most important range of topics if you are new to EasyBuild. @@ -7,7 +7,7 @@ if you are new to EasyBuild. and hereby the most common command line options are being documented. -## Specifying what to install +## Specifying what to install {: #specifying_easyconfigs } To instruct EasyBuild which software packages it should build/install and which build parameters it should use, one or more [*easyconfig files*](terminology.md#easyconfig-files) must be specified. @@ -37,7 +37,7 @@ robot search path (see [robot search path](#searching-for-easyconfigs-the-robot- part of the active EasyBuild installation for *explicitly specified* easyconfig files. -### By providing a single easyconfig file +### By providing a single easyconfig file {: #specifying_easyconfigs_single } The most basic usage is to simply provide the name of an easyconfig file to the `eb` command. EasyBuild will (try and) locate the easyconfig file, and perform the installation as specified by that easyconfig file. @@ -92,7 +92,7 @@ This is a crucial design aspect, since the dependency resolution mechanism (see see [Configuration](configuration.md) -### Via command line options +### Via command line options {: #specifying_easyconfigs_command_line } An alternative approach is to only use command line options to specify which software to build. Refer to the `Software search and build options` section in the `eb --help` output for an overview @@ -113,7 +113,7 @@ $ eb --software-name=HPL --toolchain=foss,2021b At this point, a module `HPL/2.3-foss-2021b` should have been installed. -### By providing a set of easyconfig files +### By providing a set of easyconfig files {: #specifying_easyconfigs_set_of_easyconfigs } Multiple easyconfig files can be provided as well, either directly or by specifying a directory that contains easyconfig files. @@ -192,7 +192,7 @@ This is EasyBuild 4.6.2 (framework: 4.6.2, easyblocks: 4.6.2) on host example.lo Asking EasyBuild to print own its version is a quick way to ensure that `$PYTHONPATH` is set up correctly, so that the entire EasyBuild installation (framework and easyblocks) is available. -### List of known toolchains +### List of known toolchains {: #list_toolchains } *(`--list-toolchains`)* @@ -201,7 +201,7 @@ For an overview of known [toolchains](terminology.md#toolchains), use `eb --list The complete table of available toolchains is available [here](version-specific/eb-list-toolchains.md). -### List of available easyblocks +### List of available easyblocks {: #list_easyblocks } *(`--list-easyblocks`)* @@ -223,7 +223,7 @@ To see more details about the available easyblocks, i.e., in which Python module located, use `--list-easyblocks=detailed`. -## All available easyconfig parameters +## All available easyconfig parameters {: #avail_easyconfig_params } *(`--avail-easyconfig-params` / `-a`)* @@ -262,7 +262,7 @@ eb HPL-2.3-foss-2021b.eb --debug Debug log files are significantly larger than non-debug logs, beware! -### Rebuild installation +### Rebuild installation {: #rebuild_option } *(`--rebuild`)* @@ -283,7 +283,7 @@ Use `eb --rebuild` to rebuild a given easyconfig/module. use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). -### Forced reinstallation +### Forced reinstallation {: #force_option } *(`--force` / `-f`)* @@ -305,7 +305,7 @@ The behavior of `--force` is the same as `--rebuild` and `--ignore-osdeps`. use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). -### Searching for easyconfigs +### Searching for easyconfigs {: #searching_for_easyconfigs } *(`--search` / `-S`)* @@ -359,7 +359,7 @@ CFGS1=/home/example/easybuild-easyconfigs/easybuild/easyconfigs/m/matplotlib ``` -## Enabling dependency resolution +## Enabling dependency resolution {: #use_robot } *(`--robot` / `-r` and `--robot-paths`)* @@ -405,7 +405,7 @@ by the dependency graph. Missing modules for dependencies (use --robot?): zlib/1.2.12 ``` -## Searching for easyconfigs: the robot search path +## Searching for easyconfigs: the robot search path {: #robot_search_path } For each dependency that does not have a matching module installed yet, EasyBuild requires a corresponding easyconfig file. If no such easyconfig file was specified on the `eb` command line, the dependency resolution mechanism will try @@ -452,7 +452,7 @@ by the `$PYTHONPATH` environment variable) that contain a subdirectory named `ea part of the robot search path (in the order they are encountered). -### Controlling the robot search path +### Controlling the robot search path {: #controlling_robot_search_path } To control the robot search path, you can specify a (colon-separated list of) path(s) to `--robot`/`-r` and/or `--robot-paths` (or, equivalently, `$EASYBUILD_ROBOT` and/or `$EASYBUILD_ROBOT_PATHS`): @@ -507,7 +507,7 @@ For more information on using constants in EasyBuild configuration files, see between the different configuration types, see [Supported configuration types](#supported-configuration-types). -### Prepending/appending to the default robot search path +### Prepending/appending to the default robot search path {: #robot_search_path_prepend_append } Prepending or appending to the default robot search path is supported via the `--robot-paths` configuration option. @@ -546,7 +546,7 @@ Matching setup: ``` -## Getting an overview of planned installations +## Getting an overview of planned installations {: #get_an_overview } *(`--dry-run` / `-D`)* @@ -578,7 +578,7 @@ Note how the different status symbols denote distinct handling states by EasyBui will be execute can be obtained using `--extended-dry-run` or `-x`, see [Extended dry run](extended-dry-run.md). -## Getting an overview of missing installations +## Getting an overview of missing installations {: #eb_missing } *(`--missing-modules` / `-M`)* @@ -599,7 +599,7 @@ Note that dependency resolution is enabled automatically (i.e., `--robot` is implied when using `--missing-modules` or `-M`). -## Tweaking existing easyconfig files +## Tweaking existing easyconfig files {: #tweaking_easyconfigs_using_try } *(`--try-*`)* diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index bada1a8a23..17fdde536c 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -1,7 +1,7 @@ -# Writing easyconfig files +# Writing easyconfig files {: #writing_easyconfig_files } -## What is an easyconfig (file)? +## What is an easyconfig (file)? {: #what_is_an_easyconfig } !!! warning From ed53516f0b81c17d11d4ba35a11b761fef38399c Mon Sep 17 00:00:00 2001 From: James Carpenter Date: Fri, 4 Nov 2022 22:27:23 +0000 Subject: [PATCH 4/7] additional internal link --- docs/terminology.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/terminology.md b/docs/terminology.md index 473577549b..af3d425e0c 100644 --- a/docs/terminology.md +++ b/docs/terminology.md @@ -2,6 +2,8 @@ ## Easyconfig files {: #easyconfig_files } +## Toolchains {: #toolchains } + !!! warning This page will soon replace . From d65acbd7b1de018dc472d3d90233d614bf29b48d Mon Sep 17 00:00:00 2001 From: James Carpenter Date: Fri, 4 Nov 2022 22:27:56 +0000 Subject: [PATCH 5/7] update links to new format --- docs/using-easybuild.md | 56 ++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/using-easybuild.md b/docs/using-easybuild.md index 391c425967..201105d1f3 100644 --- a/docs/using-easybuild.md +++ b/docs/using-easybuild.md @@ -10,25 +10,25 @@ and hereby the most common command line options are being documented. ## Specifying what to install {: #specifying_easyconfigs } To instruct EasyBuild which software packages it should build/install and which build parameters it should use, -one or more [*easyconfig files*](terminology.md#easyconfig-files) must be specified. +one or more [*easyconfig files*][easyconfig_files] must be specified. This can be done in a number of ways: -* [by providing a single easyconfig file](#by-providing-a-single-easyconfig-file) -* [via command line options](#via-command-line-options) -* [by providing a set of easyconfig files](#by-providing-a-set-of-easyconfig-files) -* [using `eb --from-pr`](integration-with-github.md#using-easyconfigs-from-pull-requests) +* [by providing a single easyconfig file][specifying_easyconfigs_single] +* [via command line options][specifying_easyconfigs_command_line] +* [by providing a set of easyconfig files][specifying_easyconfigs_set_of_easyconfigs] +* [using `eb --from-pr`][github_from_pr] Whenever EasyBuild searches for *explicitly specified* easyconfig files, it considers a couple of locations, i.e. (in order of preference): * (i) the local working directory -* (ii) the robot search path (see [robot search path](#searching-for-easyconfigs-the-robot-search-path)) +* (ii) the robot search path (see [robot search path][robot_search_path]) * (iii) the path to easyconfig files that are part of the active EasyBuild installation (which is included in the default robot search path) These locations are only considered for easyconfig files that are specified only by filename or using a relative path, *not* for easyconfig files that are specified via an absolute path. The dependencies are resolved using the -robot search path (see [robot search path](#searching-for-easyconfigs-the-robot-search-path)). +robot search path (see [robot search path][robot_search_path]). !!! note @@ -83,20 +83,20 @@ $ which xhpl All easyconfig file names' suffixes are `.eb` and follow format: `--` -This is a crucial design aspect, since the dependency resolution mechanism (see [robot]()) relies upon this convention. +This is a crucial design aspect, since the dependency resolution mechanism (see [robot][use_robot]) relies upon this convention. !!! tip You may wish to modify the installation prefix (e.g., using `--prefix` or by defining `$EASYBUILD_PREFIX`), in order to redefine the build/install/source path prefix to be used; default value is: `$HOME/.local/easybuild`; - see [Configuration](configuration.md) + see [Configuration][configuring_easybuild] ### Via command line options {: #specifying_easyconfigs_command_line } An alternative approach is to only use command line options to specify which software to build. Refer to the `Software search and build options` section in the `eb --help` output for an overview -of the available command line options for this purpose (see also [`--help`](#command-line-help)). +of the available command line options for this purpose (see also [`--help`][basic_usage_help]). Here is how to build and install last version of HPL (that EasyBuild is aware of) using the `foss/2021b` toolchain: @@ -196,22 +196,22 @@ This is EasyBuild 4.6.2 (framework: 4.6.2, easyblocks: 4.6.2) on host example.lo *(`--list-toolchains`)* -For an overview of known [toolchains](terminology.md#toolchains), use `eb --list-toolchains`. +For an overview of known [toolchains][toolchains], use `eb --list-toolchains`. -The complete table of available toolchains is available [here](version-specific/eb-list-toolchains.md). +The complete table of available toolchains is available [here][toolchains_table]. ### List of available easyblocks {: #list_easyblocks } *(`--list-easyblocks`)* -You can obtain a list of available [easyblocks](terminology.md#easyblocks) via `--list-easyblocks`. +You can obtain a list of available [easyblocks][easyblocks] via `--list-easyblocks`. The `--list-easyblocks` command line option prints the easyblocks in a hierarchical way, showing the inheritance patterns, with the "base" easyblock class `EasyBlock` on top. Software-specific easyblocks have a name that starts with `EB_`; the ones that do not are generic easyblocks. -(see [easyblocks](terminology.md#easyblocks) for the distinction between both types of easyblocks). +(see [easyblocks][easyblocks] for the distinction between both types of easyblocks). For example, a list of easyblocks can be obtained with: @@ -231,7 +231,7 @@ EasyBuild provides a significant amount of easyconfig parameters. An overview of all available easyconfig parameters can be obtained via `eb --avail-easyconfig-params`, or `eb -a` for short. -See [available easyconfig parameters](version-specific/easyconfig-parameters.md) for more information, +See [available easyconfig parameters][easyconfig_params] for more information, the supported parameters are a very rich set. Combine -a with `--easyblock`/`-e` to include parameters that are specific to a particular easyblock. For example: @@ -240,7 +240,7 @@ Combine -a with `--easyblock`/`-e` to include parameters that are specific to a eb -a -e EB_WRF ``` -If you want to see the full output of running this command, see [here](version-specific/eb-a-wrf.md). +If you want to see the full output of running this command, see [here][eb_a_e_EB_WRF]. ### Enable debug logging @@ -275,12 +275,12 @@ Use `eb --rebuild` to rebuild a given easyconfig/module. !!! tip Combine `--rebuild` with `--dry-run` to get a good view on which installations will be rebuilt - (see also [Getting an overview of planned installations](#getting-an-overview-of-planned-installations)). + (see also [Getting an overview of planned installations][get_an_overview]). !!! note To create a backup of existing modules that are regenerated when `--rebuild` is used, - use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). + use `--backup-modules` (see also [Backing up of existing modules][backup_modules]). ### Forced reinstallation {: #force_option } @@ -297,12 +297,12 @@ The behavior of `--force` is the same as `--rebuild` and `--ignore-osdeps`. !!! tip Combine `--force` with `--dry-run` to get a good view on which installations will be forced - (see also [Getting an overview of planned installations](#getting-an-overview-of-planned-installations)). + (see also [Getting an overview of planned installations][get_an_overview]). !!! note To create a backup of existing modules that are regenerated when `--rebuild` is used, - use `--backup-modules` (see also [Backing up of existing modules](backup-modules.md)). + use `--backup-modules` (see also [Backing up of existing modules][backup_modules]). ### Searching for easyconfigs {: #searching_for_easyconfigs } @@ -311,7 +311,7 @@ The behavior of `--force` is the same as `--rebuild` and `--ignore-osdeps`. Searching for available easyconfig files can be done using the `--search` (long output) and `-S` (short output) command line options. All easyconfig files available in the robot search path are considered -(see [robot search path](#searching-for-easyconfigs-the-robot-search-path)), and searching is done *case-insensitive*. +(see [robot search path][robot_search_path]), and searching is done *case-insensitive*. For example, to see which easyconfig files are available for the software package named `Mesquite`: @@ -411,7 +411,7 @@ For each dependency that does not have a matching module installed yet, EasyBuil file. If no such easyconfig file was specified on the `eb` command line, the dependency resolution mechanism will try to locate one in the `robot search path`. -Searching for easyconfigs is done based on filename (see also [What is an easyconfig (file)?](#what-is-an-easyconfig-file)), +Searching for easyconfigs is done based on filename (see also [What is an easyconfig (file)?][what_is_an_easyconfig]), with filenames being derived from the dependency specification (i.e. software name/version, toolchain and version suffix). For each entry in the robot search path, a couple of different filepaths are considered, mostly determined by the software name. @@ -476,19 +476,19 @@ These two configuration options each serve a particular purpose, and together de * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed; * **note**: setting this configuration option implies redefining the default robot search path, unless a prepending/appending list of paths is specified, see - [Prepending/appending to the default robot search path](#prependingappending-to-the-default-robot-search-path); + [Prepending/appending to the default robot search path][robot_search_path_prepend_append]; For both options, the list of paths should be specified as a colon-separated (`:`) list. By combining `--robot` and `--robot-paths` using the different configuration types (see also -[Supported configuration types](#supported-configuration-types)), you have full control over the robot search path: which paths are included, +[Supported configuration types][configuration_types]), you have full control over the robot search path: which paths are included, the order of those paths, whether or not the easyconfig files that are part of the EasyBuild installation should be considered, etc. A constant named `DEFAULT_ROBOT_PATHS` is available that can be used (only) in EasyBuild configuration files to refer to the default robot search path, i.e. the path to the easyconfigs that are part of the EasyBuild installation. For more information on using constants in EasyBuild configuration files, see -[Templates and constants supported in configuration files](#templates-and-constants-supported-in-configuration-files). +[Templates and constants supported in configuration files][configuration_file_templates_constants]. !!! tip @@ -504,7 +504,7 @@ For more information on using constants in EasyBuild configuration files, see the value in `$EASYBUILD_ROBOT`, `$EASYBUILD_ROBOT_PATHS` *overrides* the `robot-paths` specification in an EasyBuild configuration file, etc. Of course, the value specified for `--robot` does not directly affect the value specified for `--robot-paths`, on any configuration level, and vice versa. For more information on the relation - between the different configuration types, see [Supported configuration types](#supported-configuration-types). + between the different configuration types, see [Supported configuration types][configuration_types]. ### Prepending/appending to the default robot search path {: #robot_search_path_prepend_append } @@ -531,7 +531,7 @@ For example, say we want to configure EasyBuild to behave as follows w.r.t. the Matching setup: * satisfy (i) using `robot-paths` in one of the active EasyBuild configuration files (see also - [List of configuration files](#list-of-used-configuration-files)): + [List of configuration files][list_of_configuration_files]): ```ini [basic] @@ -575,7 +575,7 @@ Note how the different status symbols denote distinct handling states by EasyBui !!! note Since EasyBuild v2.4.0, a detailed overview of the build and install procedure that EasyBuild - will be execute can be obtained using `--extended-dry-run` or `-x`, see [Extended dry run](extended-dry-run.md). + will be execute can be obtained using `--extended-dry-run` or `-x`, see [Extended dry run][extended_dry_run]. ## Getting an overview of missing installations {: #eb_missing } From 1144d45e1ce9cf0d792e797abf77f75e41dbba0a Mon Sep 17 00:00:00 2001 From: James Carpenter Date: Sat, 5 Nov 2022 23:17:28 +0000 Subject: [PATCH 6/7] fix internal link errors --- docs/hooks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hooks.md b/docs/hooks.md index 69984672df..c350c95e95 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -25,7 +25,7 @@ you only need to specify the location of the Python module (`*.py`) that impleme This is done via the `--hooks` configuration option (or equivalently via the `$EASYBUILD_HOOKS` environment variable, or via `hooks = ...` -in an EasyBuild configuration file, see also [Configuring EasyBuild][configuring-easybuild]). +in an EasyBuild configuration file, see also [Configuring EasyBuild][configuring_easybuild]). For example: @@ -229,7 +229,7 @@ See also [Example hook to inject a custom patch file](#inject-a-custom-patch-fil EasyBuild archives the easyconfig file that was used for a particular installation: A copy is stored both in the `easybuild` subdirectory of the software installation -directory and in the easyconfigs repository (see [Easyconfigs repository][configuration-easyconfigs-repository]). +directory and in the easyconfigs repository (see [Easyconfigs repository][easyconfigs_repo]). If any changes were made to the easyconfig file via hooks, these changes will *not* be reflected in these copies. From 34a71fbde03a7586040117436cfc3a72fd040057 Mon Sep 17 00:00:00 2001 From: James Carpenter Date: Sat, 5 Nov 2022 23:17:49 +0000 Subject: [PATCH 7/7] define internal links --- docs/extended-dry-run.md | 2 +- docs/terminology.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/extended-dry-run.md b/docs/extended-dry-run.md index ed097897f6..9686f1175f 100644 --- a/docs/extended-dry-run.md +++ b/docs/extended-dry-run.md @@ -1,4 +1,4 @@ -# Extended dry run +# Extended dry run {: #extended_dry_run } !!! warning This page will soon replace . diff --git a/docs/terminology.md b/docs/terminology.md index af3d425e0c..14d956eb05 100644 --- a/docs/terminology.md +++ b/docs/terminology.md @@ -4,6 +4,8 @@ ## Toolchains {: #toolchains } +## Easyblocks {: #easyblocks } + !!! warning This page will soon replace .