From 953d30f2d296222f72061f214649eb551739da72 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 13:46:54 +0100 Subject: [PATCH 01/10] Document how to update toolchains for existing easyconfigs --- docs/writing-easyconfig-files.md | 52 ++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 23e9d36d2c..89b48ce7fb 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -940,6 +940,58 @@ Example: eb WRF-3.5.1-ictce-5.3.0-dmpar.eb --try-toolchain=intel,2014b -r ``` +## Updating existing easyconfigs for another toolchain + +While `try-toolchain` is a convenient way of testing existing software versions with another toolchain +you rarely want this behavior. +Usually when using a newer toolchain you also want to use newer versions of (at least) the dependencies. +Often there are already at least easyconfigs available for the dependencies in the new toolchain, +and especially if those are already installed you should use those as dependencies for the updated easyconfig. + +So the process of updating an easyconfig to a newer toolchain version could look like: + +1. Copy the easyconfig and change the toolchain version +1. Usually, especially when updating to the latest toolchain, + you should search for the most recent version of the software too and use that. +1. For each listed dependency find the easyconfig for the new toolchain version. + Here you need to take [toolchain hierarchy](common-toolchains.md#toolchains_diagram) into account. + I.e. a dependency of an easyconfig for the `foss` toolchain might use the `GCC` or `GCCcore` toolchain. + Check the toolchain easyconfig file (e.g. `foss-2023b.eb`) to find the versions of the sub-toolchain. + For very recent toolchains you can also check the `develop` branch of the [easyconfigs git repository](https://github.com/easybuilders/easybuild-easyconfigs) and open PRs. +1. If an easyconfig of the dependency for the new toolchain already exists use its version in your new easyconfig. + Otherwise, you need to create a new easyconfig for the new toolchain version based on that of another toolchain version. + The steps are the same and need to also be repeated for each dependency of this dependency. +1. Finally, build and test your easyconfig and consider [contributing](contributing.md#contributing-easyconfig-files--contributing_easyconfigs) it. + +Especially when creating easyconfigs for dependencies it is a good idea to put all of them into a new folder. +To allow EasyBuild to find those, pass its path via `easybuild --robot `. + +This manual process can be partially automated: + +1. Create a new folder for the new easyconfig and its dependencies. +1. Run `easybuild --try-toolchain-version= --try-update-deps --copy-ec `. + This will update the toolchain version, search for the versions of dependencies in the new toolchain and use them if available. + If a dependency for then new toolchain is not available this will create a new easyconfig for this toolchain + based on an existing one for another toolchain (version). + All newly create easyconfigs will be copied to the specified folder. +1. Optionally, but recommended, update the version of the software for each created easyconfig to the latest available one. +1. **Important**: Verify the newly created easyconfigs by comparing each of them against the existing one for that software. + Only the version of the software, toolchain and dependencies should have been changed. + EasyBuild often replaces local variables or templates by their value when updating easyconfigs. + It might be easier to use those easyconfigs just for getting the versions of each dependency required + and still copy & update the easyconfigs from existing ones manually. + +When updating a software version the patches in the easyconfig need to be checked too. +If they don't apply they might not be required anymore, when the upstream software fixed the issue, and can be removed. +Otherwise, they need to be adjusted for the changes in the source code. + +Also keep in mind that changing the version of the software and/or the toolchain might reveal new bugs or incompatibilities. +Some can be fixed or worked around by patches, often even with help from the maintainers of the software. +But some software is simply not compatible with specific versions of compilers or other software. +Feel free to ask for help in the [EasyBuild Slack channel](https://easybuild.io/join-slack). +Also filing an issue or bug report against the software in question helps in getting more information or the bug being fixed in later versions. +This fix can then also be applied to the earlier version of necessary. + ## Dynamic values for easyconfig parameters {: #easyconfig_param_templates } String templates are completed using the value of particular easyconfig From 4eb42745cf62a4b5a6b49267453c6a2d3e255f18 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 14:02:12 +0100 Subject: [PATCH 02/10] Add notes about Python packages --- docs/writing-easyconfig-files.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 89b48ce7fb..3448605075 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -985,6 +985,18 @@ When updating a software version the patches in the easyconfig need to be checke If they don't apply they might not be required anymore, when the upstream software fixed the issue, and can be removed. Otherwise, they need to be adjusted for the changes in the source code. +For updates to easyconfigs containing Python packages (`PythonBundle`) you should check each of them if there is a newer version and if it is still required. +The latest version of a single Python package can usually be found on [PyPI](https://pypi.org). +To find the list of Python dependencies for another Python package to be installed by an easyconfig you can use the `findPythonDeps` script +distributed with EasyBuild. +This will also output the latest, compatible version of the found packages. +A good approach is to start from an easyconfig just containing the dependency on Python but no Python packages and then run the `findPythonDeps`. +For every Python package displayed by the script first search for a suitable easyconfig containing that package and add it as a dependency one by one. +That might require updating it from another toolchain. +Then run the script again to honor Python packages from that dependency and its dependencies. +Repeat until none of the remaining packages displayed are in any easyconfig and/or unique enough to the specific software that an own easyconfig would be useful. +Those can then be added to the `ext_list` of the easyconfig. + Also keep in mind that changing the version of the software and/or the toolchain might reveal new bugs or incompatibilities. Some can be fixed or worked around by patches, often even with help from the maintainers of the software. But some software is simply not compatible with specific versions of compilers or other software. From a5aa9915c1ee4f4a5b7fd8f815667d344b717b76 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 16:13:09 +0100 Subject: [PATCH 03/10] Apply suggestions from code review Co-authored-by: ocaisa --- docs/writing-easyconfig-files.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 3448605075..54ecf4a5f1 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -958,20 +958,20 @@ So the process of updating an easyconfig to a newer toolchain version could look I.e. a dependency of an easyconfig for the `foss` toolchain might use the `GCC` or `GCCcore` toolchain. Check the toolchain easyconfig file (e.g. `foss-2023b.eb`) to find the versions of the sub-toolchain. For very recent toolchains you can also check the `develop` branch of the [easyconfigs git repository](https://github.com/easybuilders/easybuild-easyconfigs) and open PRs. -1. If an easyconfig of the dependency for the new toolchain already exists use its version in your new easyconfig. - Otherwise, you need to create a new easyconfig for the new toolchain version based on that of another toolchain version. - The steps are the same and need to also be repeated for each dependency of this dependency. +1. If an easyconfig for the listed dependency with the new toolchain already exists use its version in your new easyconfig. + Otherwise, you need to create a new easyconfig for the listed dependency with the new toolchain version (based on that of another toolchain version). + The steps are the same and also need to be repeated for each dependency of this dependency. 1. Finally, build and test your easyconfig and consider [contributing](contributing.md#contributing-easyconfig-files--contributing_easyconfigs) it. Especially when creating easyconfigs for dependencies it is a good idea to put all of them into a new folder. -To allow EasyBuild to find those, pass its path via `easybuild --robot `. +To allow EasyBuild to find these easyconfigs in your folder, pass its path via `eb --robot ...`. This manual process can be partially automated: 1. Create a new folder for the new easyconfig and its dependencies. 1. Run `easybuild --try-toolchain-version= --try-update-deps --copy-ec `. This will update the toolchain version, search for the versions of dependencies in the new toolchain and use them if available. - If a dependency for then new toolchain is not available this will create a new easyconfig for this toolchain + If a dependency with the new toolchain is not available this will create a new easyconfig for this toolchain based on an existing one for another toolchain (version). All newly create easyconfigs will be copied to the specified folder. 1. Optionally, but recommended, update the version of the software for each created easyconfig to the latest available one. From 16d19bc658bb9f97626a4f270a48e94867b1a072 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 17:15:38 +0100 Subject: [PATCH 04/10] Add --experimental to --try-update-deps Co-authored-by: ocaisa --- docs/writing-easyconfig-files.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 54ecf4a5f1..6cebdbd874 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -969,7 +969,7 @@ To allow EasyBuild to find these easyconfigs in your folder, pass its path via ` This manual process can be partially automated: 1. Create a new folder for the new easyconfig and its dependencies. -1. Run `easybuild --try-toolchain-version= --try-update-deps --copy-ec `. +1. Run `eb --try-toolchain-version= --try-update-deps --experimental --copy-ec `. This will update the toolchain version, search for the versions of dependencies in the new toolchain and use them if available. If a dependency with the new toolchain is not available this will create a new easyconfig for this toolchain based on an existing one for another toolchain (version). From 5ca19563119c2b4e06b057fc602ad0a83ec8b8a5 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 17:23:26 +0100 Subject: [PATCH 05/10] Small addition to replaced variables and findPythonDeps --- docs/writing-easyconfig-files.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 6cebdbd874..0cc3c11a9a 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -977,7 +977,8 @@ This manual process can be partially automated: 1. Optionally, but recommended, update the version of the software for each created easyconfig to the latest available one. 1. **Important**: Verify the newly created easyconfigs by comparing each of them against the existing one for that software. Only the version of the software, toolchain and dependencies should have been changed. - EasyBuild often replaces local variables or templates by their value when updating easyconfigs. + EasyBuild often replaces local variables or templates like `name` or `%(version)s` by their value when updating easyconfigs. + This is undesirable as the values might become out of sync with what they are supposed to be, e.g. after changing the version variable. It might be easier to use those easyconfigs just for getting the versions of each dependency required and still copy & update the easyconfigs from existing ones manually. @@ -990,7 +991,7 @@ The latest version of a single Python package can usually be found on [PyPI](htt To find the list of Python dependencies for another Python package to be installed by an easyconfig you can use the `findPythonDeps` script distributed with EasyBuild. This will also output the latest, compatible version of the found packages. -A good approach is to start from an easyconfig just containing the dependency on Python but no Python packages and then run the `findPythonDeps`. +A good approach is to start from an easyconfig just containing the dependency on Python but no Python packages and then run e.g. `findPythonDeps.py --ec foo-1.2.3.eb foo==1.2.3`. For every Python package displayed by the script first search for a suitable easyconfig containing that package and add it as a dependency one by one. That might require updating it from another toolchain. Then run the script again to honor Python packages from that dependency and its dependencies. From 7e51b3432ffb1d7a0bf48ac89f93483efa5b4f0c Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Wed, 8 Jan 2025 17:27:22 +0100 Subject: [PATCH 06/10] Refer to --help --- docs/writing-easyconfig-files.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 0cc3c11a9a..7c29e1b893 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -997,6 +997,7 @@ That might require updating it from another toolchain. Then run the script again to honor Python packages from that dependency and its dependencies. Repeat until none of the remaining packages displayed are in any easyconfig and/or unique enough to the specific software that an own easyconfig would be useful. Those can then be added to the `ext_list` of the easyconfig. +More information about that script can be found at the top of the script file and via `findPythonDeps.py --help`. Also keep in mind that changing the version of the software and/or the toolchain might reveal new bugs or incompatibilities. Some can be fixed or worked around by patches, often even with help from the maintainers of the software. From 1d39c129fbd39d4e24dcc27f0a171d5800714aaa Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 9 Jan 2025 10:39:15 +0100 Subject: [PATCH 07/10] Add path to findPythonDeps --- docs/writing-easyconfig-files.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 7c29e1b893..83b3ffd10f 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -990,6 +990,7 @@ For updates to easyconfigs containing Python packages (`PythonBundle`) you shoul The latest version of a single Python package can usually be found on [PyPI](https://pypi.org). To find the list of Python dependencies for another Python package to be installed by an easyconfig you can use the `findPythonDeps` script distributed with EasyBuild. +For a typical installation it can be run using `$(dirname $(which eb))/../easybuild/scripts/findPythonDeps.py` which automatically finds the standard path for it. This will also output the latest, compatible version of the found packages. A good approach is to start from an easyconfig just containing the dependency on Python but no Python packages and then run e.g. `findPythonDeps.py --ec foo-1.2.3.eb foo==1.2.3`. For every Python package displayed by the script first search for a suitable easyconfig containing that package and add it as a dependency one by one. From e69c6c17d037e046047bf936881ae4868dd96f1f Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 9 Jan 2025 12:40:57 +0100 Subject: [PATCH 08/10] Add note about checksums Co-authored-by: ocaisa --- docs/writing-easyconfig-files.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 83b3ffd10f..cf485747c7 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -953,6 +953,11 @@ So the process of updating an easyconfig to a newer toolchain version could look 1. Copy the easyconfig and change the toolchain version 1. Usually, especially when updating to the latest toolchain, you should search for the most recent version of the software too and use that. + When you update the version, the checksum for the softwares sources will + become incorrect and should be removed. You can automatically inject the + checksums for the new sources with `eb --inject-checksums ` + (see [checksums documentation](writing-easyconfig-files.md#common_easyconfig_param_sources_checksums) + for more details). 1. For each listed dependency find the easyconfig for the new toolchain version. Here you need to take [toolchain hierarchy](common-toolchains.md#toolchains_diagram) into account. I.e. a dependency of an easyconfig for the `foss` toolchain might use the `GCC` or `GCCcore` toolchain. From 618e1b6f324e9763e53ab82c789dcceed95b4ccc Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 9 Jan 2025 12:41:23 +0100 Subject: [PATCH 09/10] Improve hint about undesirable changes by try-update-deps Co-authored-by: ocaisa --- docs/writing-easyconfig-files.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index cf485747c7..2fea9dcc0c 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -981,11 +981,13 @@ This manual process can be partially automated: All newly create easyconfigs will be copied to the specified folder. 1. Optionally, but recommended, update the version of the software for each created easyconfig to the latest available one. 1. **Important**: Verify the newly created easyconfigs by comparing each of them against the existing one for that software. - Only the version of the software, toolchain and dependencies should have been changed. - EasyBuild often replaces local variables or templates like `name` or `%(version)s` by their value when updating easyconfigs. - This is undesirable as the values might become out of sync with what they are supposed to be, e.g. after changing the version variable. - It might be easier to use those easyconfigs just for getting the versions of each dependency required - and still copy & update the easyconfigs from existing ones manually. + Only the version of the software, toolchain and dependencies should have been changed. However, this approach + can lead to unwanted changes to formatting as well as the replacement local variables or templates (such as `name`, + `version` or `%(version)s`) by their _value_ when updating easyconfigs. + This is undesirable as the values might become out of sync with what they are supposed to be, e.g. when we update + the `version` variable the old _value_ may still exist elsewhere in the easyconfig. + Depending on the situation, it might be easier to use the newly created easyconfigs just for getting the versions of + each dependency required and still copy & update the easyconfigs from existing ones manually. When updating a software version the patches in the easyconfig need to be checked too. If they don't apply they might not be required anymore, when the upstream software fixed the issue, and can be removed. From 75dce11053a26808951b95b033d6a994808f89b0 Mon Sep 17 00:00:00 2001 From: ocaisa Date: Thu, 9 Jan 2025 12:46:39 +0100 Subject: [PATCH 10/10] Update writing-easyconfig-files.md --- docs/writing-easyconfig-files.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 2fea9dcc0c..1afa9dc97a 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -953,7 +953,7 @@ So the process of updating an easyconfig to a newer toolchain version could look 1. Copy the easyconfig and change the toolchain version 1. Usually, especially when updating to the latest toolchain, you should search for the most recent version of the software too and use that. - When you update the version, the checksum for the softwares sources will + When you update the version, the checksum for the software sources will become incorrect and should be removed. You can automatically inject the checksums for the new sources with `eb --inject-checksums ` (see [checksums documentation](writing-easyconfig-files.md#common_easyconfig_param_sources_checksums) @@ -983,9 +983,9 @@ This manual process can be partially automated: 1. **Important**: Verify the newly created easyconfigs by comparing each of them against the existing one for that software. Only the version of the software, toolchain and dependencies should have been changed. However, this approach can lead to unwanted changes to formatting as well as the replacement local variables or templates (such as `name`, - `version` or `%(version)s`) by their _value_ when updating easyconfigs. + `version` or `%(version)s`) by their *value* when updating easyconfigs. This is undesirable as the values might become out of sync with what they are supposed to be, e.g. when we update - the `version` variable the old _value_ may still exist elsewhere in the easyconfig. + the `version` variable the old *value* may still exist elsewhere in the easyconfig. Depending on the situation, it might be easier to use the newly created easyconfigs just for getting the versions of each dependency required and still copy & update the easyconfigs from existing ones manually.