enable `download_dep_fail`, `use_pip`, `sanity_pip_check` by default in `PythonPackage` easyblock #3022

boegel · 2023-10-23T07:52:22Z

Probably worth doing a couple of test builds for the easyblocks that are touched, to avoid surprises.

Easyconfigs which are currently enabling these easyconfig parameters should be cleaned up.

Non-archived easyconifgs which are not enabling these easyconfig parameters yet should be updated to set them to False (unless using pip and running "pip check" works fine for them).

fixes #2127

…honPackage easyblock

Flamefire · 2023-11-08T11:04:01Z

easybuild/easyblocks/generic/pythonpackage.py

@@ -417,7 +417,7 @@ def determine_install_command(self):
        """
        Determine install command to use.
        """
-        if self.cfg.get('use_pip', False) or self.cfg.get('use_pip_editable', False):
+        if self.cfg.get('use_pip', True) or self.cfg.get('use_pip_editable', False):


Do we still need this duplication of the default params in EB 5.x or could we just switch to self.cfg['use_pip'] and fix EasyBlocks not "inheriting" the options this block requires?

I think it's wise to keep using self.cfg.get, to avoid hard crashes in easyblock to derive from PythonPackage but do not inherit the custom easyconfig parameters from PythonPackage.
We can make sure the easyblocks we have in the central repository do so, but we don't control all easyblocks out there, and it's a small effort to avoid fallout...

The self.cfg.get is a pattern that we should use in all generic easyblocks (and even some custom ones) since they may be used as a base for other easyblocks.

I understand that motivation for EB 4.x but I wanted to suggest that such a breakage could be OK for EB 5.x. Not inheriting the parameters has several other implications such as not being able to list available parameters correctly. This introduces some kind of "hidden" parameters and puts burden on upstream EB to be careful to always use the self.cfg.get pattern and deal with potential different default values. Technically each instance of those self.cfg.get calls would need a check against the extra_parameters section that the default matches and that possibly even across easyblocks (some descendant of e.g. PythonPackage also using self.cfg.get('use_pip' would possibly need to be updated too)

@Flamefire It's a good point, and I'm happy to follow up on it, but we should do that consistently then, so let's open an issue on this.

The window of opportunity w.r.t. (potentially) breaking changes to include in EasyBuild v5.0 is slowly closing though...

Yes, I agree that it is a good point. Let's follow up on this topic in a separate issue or PR. I will merge this PR as is so we can continue testing the 5.0.x branch for EasyBuild v5.0.

boegel · 2024-01-03T16:05:15Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS wxPython-4.1.1-foss-2021a.eb
SUCCESS wxPython-4.2.0-foss-2021b.eb
SUCCESS wxPython-4.2.1-foss-2022a.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3608.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegel/cd9d4c7ce5ba190e264eddc8f4db7e2b for a full test report.

boegel · 2024-01-03T18:49:30Z

List of easyconfig files currently in 5.0.x branch (120 out of 8,665 easyconfigs) that do not set one of download_dep_fail, use_pip, or sanity_pip_check, and hence may get broken with this change:

CRPropa (CRPropa-3.1.6-foss-2020a-Python-3.8.2.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
dm-reverb (dm-reverb-0.2.0-foss-2020b.eb, dm-reverb-0.7.0-foss-2021b.eb)
- dm-reverb-0.2.0-foss-2020b.eb: pip check fails with "dm-reverb 0.2.0 requires dm-tree, which is not installed" => fixed in add missing dm-tree dependency for dm-reverb 0.2.0 easybuild-easyconfigs#19653
- dm-reverb-0.7.0-foss-2021b.eb works fine
ESMPy (ESMPy-8.0.1-foss-2020a-Python-3.8.2.eb, ESMPy-8.0.1-intel-2020b.eb)
- use_pip is explicitly disabled for ESMPy because it doesn't work for these versions => no changes needed
EZC3D (EZC3D-1.5.2-foss-2022a.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
EasyBuild (EasyBuild-3.7.1.eb, EasyBuild-3.8.0.eb, EasyBuild-3.8.1.eb, EasyBuild-3.9.0.eb, EasyBuild-3.9.1.eb, EasyBuild-3.9.2.eb, EasyBuild-3.9.3.eb, EasyBuild-3.9.4.eb, EasyBuild-4.0.0.eb, EasyBuild-4.0.1.eb, EasyBuild-4.1.0.eb, EasyBuild-4.1.1.eb, EasyBuild-4.1.2.eb, EasyBuild-4.2.0.eb, EasyBuild-4.2.1.eb, EasyBuild-4.2.2.eb, EasyBuild-4.3.0.eb, EasyBuild-4.3.1.eb, EasyBuild-4.3.2.eb, EasyBuild-4.3.3.eb, EasyBuild-4.3.4.eb, ,EasyBuild-4.4.0.eb, EasyBuild-4.4.1.eb, EasyBuild-4.4.2.eb, EasyBuild-4.5.0.eb, EasyBuild-4.5.1.eb,EasyBuild-4.5.2.eb, EasyBuild-4.5.3.eb, EasyBuild-4.5.4.eb, EasyBuild-4.5.5.eb, EasyBuild-4.6.0.eb, EasyBuild-4.6.1.eb, EasyBuild-4.6.2.eb, EasyBuild-4.7.0.eb, EasyBuild-4.7.1.eb, EasyBuild-4.7.2.eb, EasyBuild-4.8.0.eb, EasyBuild-4.8.1.eb, EasyBuild-4.8.2.eb
- partially fixed by archive EasyBuild 3.x easyconfigs easybuild-easyconfigs#19656
- EasyBuild 4.x easyconfigs work just fine on top of the changes in this PR
GenomeWorks (GenomeWorks-2021.02.2-fosscuda-2020b.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
gemmi (gemmi-0.4.5-GCCcore-10.2.0.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
ITK (ITK-5.1.2-foss-2020a-Python-3.8.2.eb, ITK-5.1.2-fosscuda-2020a-Python-3.8.2.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
JUBE (JUBE-2.4.0.eb, JUBE-2.4.1.eb, JUBE-2.4.2.eb)
- irrelevant, archiving these easyconfigs, see easyconfigs PR #19494
libxml2 (libxml2-2.10.3-GCCcore-12.2.0.eb, libxml2-2.11.4-GCCcore-12.3.0.eb, libxml2-2.11.5-GCCcore-13.2.0.eb, libxml2-2.9.10-GCCcore-10.2.0.eb, libxml2-2.9.10-GCCcore-10.3.0.eb, libxml2-2.9.10-GCCcore-11.2.0.eb, libxml2-2.9.10-GCCcore-9.3.0.eb, libxml2-2.9.13-GCCcore-11.3.0.eb)
- => custom easyblock for libxml2 derives from PythonPackage, but that is only relevant if libxml2 includes a Python dependency (which it doesn't in these easyconfigs) => no changes needed
MUSCLE3 (MUSCLE3-0.7.0-foss-2022b.eb)
- => installation on top of these changes works fine, no changes needed
numexpr (numexpr-2.7.1-foss-2020a-Python-3.8.2.eb, numexpr-2.7.1-intel-2020a-Python-3.8.2.eb, numexpr-2.8.1-foss-2021a.eb, numexpr-2.8.1-intel-2021a.eb, numexpr-2.8.4-foss-2022a.eb)
- => installation on top of these changes works fine, no changes needed
PySide2 (PySide2-5.14.2.3-GCCcore-10.2.0.eb)
- => installation with pip does not work, already sets use_pip to False explicitly => OK
PyTorch (PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb, PyTorch-1.10.0-foss-2021a.eb, PyTorch-1.10.0-fosscuda-2020b.eb, PyTorch-1.11.0-foss-2021a-CUDA-11.3.1.eb, PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb, PyTorch-1.12.0-foss-2022a.eb, PyTorch-1.12.1-foss-2021a-CUDA-11.3.1.eb, PyTorch-1.12.1-foss-2021a.eb, PyTorch-1.12.1-foss-2021b-CUDA-11.5.2.eb, PyTorch-1.12.1-foss-2021b.eb, PyTorch-1.12.1-foss-2022a-CUDA-11.7.0.eb, PyTorch-1.12.1-foss-2022a.eb, PyTorch-1.13.1-foss-2022a-CUDA-11.7.0.eb, PyTorch-1.13.1-foss-2022a.eb, PyTorch-1.13.1-foss-2022b.eb, PyTorch-1.3.1-fosscuda-2020b.eb, PyTorch-1.7.1-foss-2020b.eb, PyTorch-1.7.1-fosscuda-2020a-Python-3.8.2.eb, PyTorch-1.7.1-fosscuda-2020b.eb, PyTorch-1.8.1-foss-2020b.eb, PyTorch-1.8.1-fosscuda-2020b.eb, PyTorch-1.9.0-foss-2020b.eb, PyTorch-1.9.0-fosscuda-2020b-imkl.eb, PyTorch-1.9.0-fosscuda-2020b.eb, , PyTorch-2.0.1-foss-2022a.eb, PyTorch-2.0.1-foss-2022b.eb, PyTorch-2.1.2-foss-2022a.eb, PyTorch-2.1.2-foss-2022b.eb)
- see disable use_pip by default for PyTorch, except for recent versions (>= 2.0) #3079
PyYAML (PyYAML-3.13.eb)
- irrelevant, archiving this easyconfig, see easyconfigs PR #19494
py-cpuinfo (py-cpuinfo-5.0.0.eb)
- irrelevant, archiving this easyconfig, see easyconfigs PR #19494
pygmo (pygmo-2.16.1-foss-2020b.eb, pygmo-2.18.0-foss-2021a.eb, pygmo-2.18.0-foss-2021b.eb, pygmo-2.18.0-foss-2022a.eb)
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
pytest-cpp (pytest-cpp-2.3.0-GCCcore-11.3.0.eb)
- download_dep_fail set to False, but works fine with download_dep_fail set to True
pytest-flakefinder (pytest-flakefinder-1.1.0-GCCcore-11.3.0.eb, pytest-flakefinder-1.1.0-GCCcore-12.2.0.eb)
- download_dep_fail set to False, but works fine with download_dep_fail set to True
pytest-rerunfailures (pytest-rerunfailures-11.1-GCCcore-11.3.0.eb, pytest-rerunfailures-12.0-GCCcore-12.2.0.eb, pytest-rerunfailures-12.0-GCCcore-12.3.0.eb)
- download_dep_fail set to False, but works fine with download_dep_fail set to True
pytest-shard (pytest-shard-0.1.2-GCCcore-11.3.0.eb, pytest-shard-0.1.2-GCCcore-12.2.0.eb)
- download_dep_fail set to False, but works fine with download_dep_fail set to True
ReFrame (ReFrame-2.18.eb, ReFrame-2.19.eb, ReFrame-2.20.eb, ReFrame-2.21.eb)
- irrelevant, archiving these easyconfigs, see easyconfigs PR #19494
SCGid (SCGid-0.9b0-foss-2021b.eb)
- => installation on top of these changes works fine, no changes needed
setuptools (setuptools-41.0.1-py3.eb)
- irrelevant, archiving these easyconfigs, see easyconfigs PR #19494
TensorRT (TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb)
- pip check fails with: uff 0.6.9 requires numpy, which is not installed => missing SciPy-bundle dependency?
- => fixed in: add missing SciPy-bundle dependency to TensorRT easyconfig easybuild-easyconfigs#19486
tmap (tmap-20220502-GCC-11.2.0.eb)
- uses CMakePythonPackage easyblock, which inherits from PythonPackage
- installation fails in install step with:
```
== FAILED: Installation ended unsuccessfully: cmd " make install  --verbose  --no-deps  --ignore-installed  --no-index " exited with exit code 2 and output:
make: unrecognized option '--verbose'
make: unrecognized option '--no-deps'
...
```
- fixing this will require some additional effort in PythonPackage, it's actually a bug: determine_install_command updates installopts in place, with the intention to use it for pip install, but then it also applies to make install which is called by CMakeMake;
- => fixed with don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080
wxPython (wxPython-4.1.1-foss-2021a.eb, wxPython-4.2.0-foss-2021b.eb, wxPython-4.2.1-foss-2022a.eb)
- wxPython easyblock already uses pip install in install_step, so not affected by use_pip;
- enabling of download_dep_fail and sanity_pip_check in PythonPackage is not a problem: installation of wxPython still works fine
- => no change needed

…pip, since PythonPackage now enables use_pip by default

boegel · 2024-01-23T08:52:27Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS CRPropa-3.1.6-foss-2020a-Python-3.8.2.eb
SUCCESS dm-reverb-0.7.0-foss-2021b.eb
SUCCESS ESMPy-8.0.1-intel-2020b.eb
SUCCESS EZC3D-1.5.2-foss-2022a.eb
SUCCESS EasyBuild-4.8.2.eb
SUCCESS GenomeWorks-2021.02.2-fosscuda-2020b.eb
SUCCESS gemmi-0.4.5-GCCcore-10.2.0.eb
SUCCESS ITK-5.1.2-foss-2020a-Python-3.8.2.eb
SUCCESS libxml2-2.9.13-GCCcore-11.3.0.eb
SUCCESS MUSCLE3-0.7.0-foss-2022b.eb
SUCCESS numexpr-2.8.4-foss-2022a.eb
SUCCESS PySide2-5.14.2.3-GCCcore-10.2.0.eb
SUCCESS pygmo-2.18.0-foss-2022a.eb
SUCCESS pytest-cpp-2.3.0-GCCcore-11.3.0.eb
SUCCESS pytest-flakefinder-1.1.0-GCCcore-12.2.0.eb
SUCCESS pytest-rerunfailures-12.0-GCCcore-12.3.0.eb
SUCCESS pytest-shard-0.1.2-GCCcore-12.2.0.eb
SUCCESS SCGid-0.9b0-foss-2021b.eb
SUCCESS TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb
SUCCESS tmap-20220502-GCC-11.2.0.eb
SUCCESS wxPython-4.2.1-foss-2022a.eb
SUCCESS matplotlib-3.8.2-gfbf-2023b.eb
SUCCESS tqdm-4.66.1-GCCcore-12.3.0.eb
SUCCESS jax-0.4.4-foss-2022a.eb
SUCCESS pybind11-2.11.1-GCCcore-13.2.0.eb
SUCCESS sympy-1.11.1-foss-2022a.eb
SUCCESS TensorFlow-2.13.0-foss-2023a.eb
SUCCESS torchvision-0.14.1-foss-2022b.eb
SUCCESS PyTorch-2.1.2-foss-2022b.eb

Build succeeded for 29 out of 29 (29 easyconfigs in total)
node3604.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegel/cc1f3b66d5e983c73a28c85874d114c2 for a full test report.

SebastianAchilles · 2024-01-23T21:05:12Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-4.9.0.eb
SUCCESS Pillow-10.0.0-GCCcore-12.3.0.eb
SUCCESS numexpr-2.8.4-foss-2022a.eb
SUCCESS pybind11-2.11.1-GCCcore-13.2.0.eb
SUCCESS sympy-1.12-gfbf-2023a.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
SUCCESS matplotlib-3.8.2-gfbf-2023b.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
jscclxc1.int.jsc-clx.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, Intel Xeon Processor (Cascadelake) (cascadelake), Python 3.9.18
See https://gist.github.com/SebastianAchilles/2cadf7dc19b1e052601e31d1955962a8 for a full test report.

SebastianAchilles

lgtm

SebastianAchilles · 2024-01-24T19:50:43Z

Going in, thanks @boegel!

enable download_dep_fail, use_pip, sanity_pip_check by default in Pyt…

d6f0cd7

…honPackage easyblock

boegel added change EasyBuild-5.0 EasyBuild 5.0 labels Oct 23, 2023

boegel added this to the 5.0 milestone Oct 23, 2023

boegel mentioned this pull request Nov 8, 2023

Set better defaults in PythonPackage #2127

Closed

Flamefire reviewed Nov 8, 2023

View reviewed changes

SebastianAchilles mentioned this pull request Nov 20, 2023

clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail easybuilders/easybuild-easyconfigs#19265

Merged

1 task

This was referenced Jan 3, 2024

add missing SciPy-bundle dependency to TensorRT easyconfig easybuilders/easybuild-easyconfigs#19486

Merged

archive old easyconfigs that rely on PythonPackage and don't enable use_pip & co easybuilders/easybuild-easyconfigs#19494

Merged

This was referenced Jan 17, 2024

disable use_pip by default for PyTorch, except for recent versions (>= 2.0) #3079

Merged

don't change installopts easyconfig parameter value in-place in PythonPackage easyblock #3080

Merged

Merge branch '5.0.x' into pythonpackage_pip_defaults

862a700

boegel mentioned this pull request Jan 19, 2024

add missing dm-tree dependency for dm-reverb 0.2.0 easybuilders/easybuild-easyconfigs#19653

Merged

boegel added 2 commits January 19, 2024 18:02

Merge branch '5.0.x' into pythonpackage_pip_defaults

51f0cea

remove code from EasyBuildMeta easyblock to conditionally enable use_…

4b48328

…pip, since PythonPackage now enables use_pip by default

easybuilders deleted a comment from boegelbot Jan 22, 2024

SebastianAchilles self-assigned this Jan 22, 2024

Merge branch '5.0.x' into pythonpackage_pip_defaults

5834749

SebastianAchilles approved these changes Jan 24, 2024

View reviewed changes

SebastianAchilles merged commit f1221e2 into easybuilders:5.0.x Jan 24, 2024
17 checks passed

boegel deleted the pythonpackage_pip_defaults branch January 24, 2024 20:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable `download_dep_fail`, `use_pip`, `sanity_pip_check` by default in `PythonPackage` easyblock #3022

enable `download_dep_fail`, `use_pip`, `sanity_pip_check` by default in `PythonPackage` easyblock #3022

boegel commented Oct 23, 2023 •

edited

Flamefire Nov 8, 2023

boegel Nov 8, 2023

Flamefire Nov 8, 2023

boegel Jan 24, 2024

SebastianAchilles Jan 24, 2024

boegel commented Jan 3, 2024

boegel commented Jan 3, 2024 •

edited

boegel commented Jan 23, 2024

SebastianAchilles commented Jan 23, 2024

SebastianAchilles left a comment

SebastianAchilles commented Jan 24, 2024

enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock #3022

enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock #3022

Conversation

boegel commented Oct 23, 2023 • edited

Flamefire Nov 8, 2023

Choose a reason for hiding this comment

boegel Nov 8, 2023

Choose a reason for hiding this comment

Flamefire Nov 8, 2023

Choose a reason for hiding this comment

boegel Jan 24, 2024

Choose a reason for hiding this comment

SebastianAchilles Jan 24, 2024

Choose a reason for hiding this comment

boegel commented Jan 3, 2024

Overview of tested easyconfigs (in order)

boegel commented Jan 3, 2024 • edited

boegel commented Jan 23, 2024

Overview of tested easyconfigs (in order)

SebastianAchilles commented Jan 23, 2024

Overview of tested easyconfigs (in order)

SebastianAchilles left a comment

Choose a reason for hiding this comment

SebastianAchilles commented Jan 24, 2024

enable `download_dep_fail`, `use_pip`, `sanity_pip_check` by default in `PythonPackage` easyblock #3022

enable `download_dep_fail`, `use_pip`, `sanity_pip_check` by default in `PythonPackage` easyblock #3022

boegel commented Oct 23, 2023 •

edited

boegel commented Jan 3, 2024 •

edited