Patch `ctypes` for Python installations when filtering `LD_LIBRARY_PATH` + fix path to `ldconfig` when using alternate sysroot #3352

ocaisa · 2024-06-07T10:36:05Z

ctypes relies on LD_LIBRARY_PATH and doesn't respect rpath linking, this is a workaround for the EasyBuild context for this setup.

ocaisa · 2024-06-07T13:13:30Z

Patching seems to work (edit: but not as originally intended):

alanc@~/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks(ctypes_python)$ eb --include-easyblocks-from-pr 3352 Python-3.11.5-GCCcore-13.2.0.eb --rebuild --stop patch
== Temporary log file in case of crash /tmp/eb-7eym3fl5/easybuild-m8zrpch1.log
  >> download succeeded: https://github.com/easybuilders/easybuild-easyblocks/archive/develop.tar.gz
  >> running command:
        [started at: 2024-06-07 15:10:31]
        [working dir: /tmp/eb-7eym3fl5/tmpnvx85rzn/easybuilders]
        [output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-rp8160c1.log]
        tar xzf /tmp/eb-7eym3fl5/tmpnvx85rzn/easybuilders/develop.tar.gz
  >> command completed: exit 0, ran in < 1s
  >> download succeeded: https://github.com/easybuilders/easybuild-easyblocks/pull/3352.diff
== easyblock python.py included from PR #3352

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* EESSI-extend/2023.06-easybuild

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

== found valid index for /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs, so using it...
== Running parse hook for Python-3.11.5-GCCcore-13.2.0.eb...
== found valid index for /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs, so using it...
== Running parse hook for GCCcore-13.2.0.eb...
== Running parse hook for GCCcore-13.2.0.eb...
== processing EasyBuild easyconfig 
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb
== building and installing Python/3.11.5-GCCcore-13.2.0...
  >> installation prefix: /home/alanc/eessi/versions/2023.06/software/linux/x86_64/generic/software/Python/3.11.5-GCCcore-13.2.0
== fetching files...
  >> sources:
  >> /tmp/alanc/easybuild/sources/p/Python/Python-3.11.5.tgz [SHA256: a12a0a013a30b846c786c010f2c19dd36b7298d888f7c4bd1581d90ce18b5e58]
  >> download succeeded: https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=59e2ce0e6b46bb47ef81b68b600ed087e14fdaad
== ... (took 1 secs)
== creating build dir, resetting environment...
  >> build dir: /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0
== Running post-ready hook...
== ... (took < 1 sec)
== unpacking...
  >> running command:
        [started at: 2024-06-07 15:10:36]
        [working dir: /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0]
        [output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-9o1mqhq7.log]
        tar xzf /tmp/alanc/easybuild/sources/p/Python/Python-3.11.5.tgz
  >> command completed: exit 0, ran in < 1s
== ... (took < 1 sec)
== patching...
== ... (took < 1 sec)
  >> running command:
        [started at: 2024-06-07 15:10:37]
        [working dir: /home/alanc/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks]
        [output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-i09p25cp.log]
        bzip2 /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/easybuild/easybuild-Python-3.11.5-20240607.151037.log
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation STOPPED successfully (took 2 secs)
== Results of the build can be found in the log file(s) /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/easybuild/easybuild-Python-3.11.5-20240607.151037.log.bz2

== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-7eym3fl5/easybuild-m8zrpch1.log* have been removed.
== Temporary directory /tmp/eb-7eym3fl5 has been removed.

alanc@~/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks(ctypes_python)$ grep LD_L /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/Python-3.11.5/Lib/ctypes/util.py
            libpath = ':'.join(filter(None, os.environ.get('LD_LIBRARY_PATH', '').split(':') + os.environ.get('LIBRARY_PATH', '').split(':')))

ocaisa · 2024-06-07T14:10:52Z

@boegelbot please test @ jsc-zen3 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"

boegelbot · 2024-06-07T14:33:08Z

@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3352 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 4318

Test results coming soon (I hope)...

- notification for comment with ID 2154930449 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

ocaisa · 2024-06-07T14:44:52Z

@boegelbot please test @ generoso EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"

boegelbot · 2024-06-07T14:50:05Z

@ocaisa: Request for testing this PR well received on login1

PR test command 'EB_PR=3352 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 13671

Test results coming soon (I hope)...

- notification for comment with ID 2154994489 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-06-07T15:06:05Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/3cad8540c8ccb38a784472ca19e53e94 for a full test report.

ocaisa · 2024-06-07T15:25:51Z

@boegelbot please test @ jsc-zen3 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"

boegelbot · 2024-06-07T15:33:08Z

@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3352 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 4321

Test results coming soon (I hope)...

- notification for comment with ID 2155065309 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-06-07T15:46:33Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/809da3be74a15531dbf6c7286c67ef06 for a full test report.

ocaisa · 2024-06-07T16:07:20Z

@boegelbot please test @ generoso EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"

easybuild/easyblocks/p/python.py

boegelbot · 2024-06-07T16:10:08Z

@ocaisa: Request for testing this PR well received on login1

PR test command 'EB_PR=3352 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 13673

Test results coming soon (I hope)...

- notification for comment with ID 2155132179 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-06-07T16:52:49Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/bbb52ea4e37e830361b159c1bb849f5c for a full test report.

boegelbot · 2024-06-07T18:15:57Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/586ae36ed6c653da99c4548e9da3cd6b for a full test report.

easybuild/easyblocks/p/python.py

ocaisa · 2024-06-10T11:27:42Z

Ok, the latest commit fixes the problem:

{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module load foss
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module load Python
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module unload EESSI-extend/2023.06-easybuild

Due to MODULEPATH changes, the following have been reloaded:
  1) Python/3.11.5-GCCcore-13.2.0

{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libmpi.so.40: cannot open shared object file: No such file or directory

However, I'm still seeing if there is a better way

easybuild/easyblocks/p/python.py

casparvl · 2024-07-26T08:56:39Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS protobuf-python-3.19.4-GCCcore-11.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/b3d931f040bc73381354d9f10f706a59 for a full test report.

casparvl · 2024-07-26T08:57:15Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS protobuf-python-4.23.0-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/fd4eb5e6123574e2a7e459252984fc7a for a full test report.

casparvl · 2024-07-26T08:57:48Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS protobuf-python-4.24.0-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/dd035ae55966e2d44d30e027fb7b6341 for a full test report.

casparvl · 2024-07-26T08:58:24Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS protobuf-python-4.25.3-GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/a3047a83f1f932dbb7b9d89216650725 for a full test report.

casparvl · 2024-07-26T09:23:40Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/5503e28e99fcee78c5fca69b23448653 for a full test report.

casparvl · 2024-07-26T09:52:32Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/df3dd3e765813ddac63670695f55fab3 for a full test report.

casparvl · 2024-07-26T09:55:55Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS python-casacore-3.5.2-foss-2023b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/3b951dcd30af89988f9580bf47b534a8 for a full test report.

casparvl · 2024-07-26T10:02:31Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS python-igraph-0.10.6-foss-2022b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/ef0b07a8d1a304db77e7dd13a944110e for a full test report.

casparvl · 2024-07-26T10:24:05Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS python-libsbml-5.20.2-foss-2023b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/4320e7e29660569454bcab14e065e838 for a full test report.

casparvl · 2024-07-26T10:24:20Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS python-xxhash-3.4.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/4a1a03af852478b86f3361b12f551cc8 for a full test report.

casparvl · 2024-07-26T11:28:41Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2022.05-foss-2022a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/0148230a9620c61e3a3a2d5e174f169e for a full test report.

casparvl · 2024-07-26T12:08:52Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/d93813bc3a2d0a9797825b19d237a265 for a full test report.

ocaisa · 2024-07-26T12:44:24Z

This should also fix the scenario described in #2183 (I think)

casparvl · 2024-07-26T13:19:11Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/5c9e9abd2b26e2aba779d10217b4fee6 for a full test report.

casparvl · 2024-07-26T14:31:07Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/9ad8bdbe0df695c45b92f11be86b0390 for a full test report.

casparvl · 2024-07-26T14:31:39Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS spglib-python-2.0.0-foss-2022a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/18bff28d9efb3b9d69e89f653363e2db for a full test report.

casparvl · 2024-07-26T14:32:06Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS spglib-python-2.0.2-gfbf-2022b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/02e2fd31548ebd919d9ced0bfa3b11de for a full test report.

casparvl · 2024-07-26T14:45:36Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS wxPython-4.2.1-foss-2022a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/8498e30ed1f417940c86ff7d6f70972e for a full test report.

casparvl · 2024-07-26T15:01:19Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS Z3-4.10.2-GCCcore-11.3.0-Python-3.10.4.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/49c387f541bc559c1cab315c7822e1c1 for a full test report.

casparvl · 2024-07-26T15:17:51Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS Z3-4.12.2-GCCcore-12.2.0-Python-3.10.8.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/f44a10916125ca2e7e4878298ddea648 for a full test report.

casparvl · 2024-07-26T17:47:23Z

In case anyone wants to replicate my tests, I ran with an easystack file containing:

easyconfigs:
  - Python-3.10.4-GCCcore-11.3.0.eb:
  - Python-3.10.8-GCCcore-12.2.0.eb:
  - Python-3.11.3-GCCcore-12.3.0.eb:
  - Python-3.11.5-GCCcore-13.2.0.eb:
  - Biopython-1.79-foss-2022a.eb:
  - Biopython-1.81-foss-2022b.eb:
  - Boost.Python-1.79.0-GCC-11.3.0.eb:
  - Boost.Python-1.83.0-GCC-13.2.0.eb:
  - Brotli-python-1.0.9-GCCcore-11.3.0.eb:
#  - flatbuffers-python-2.0-GCCcore-10.3.0.eb:
  - flatbuffers-python-23.5.26-GCCcore-12.3.0.eb:
#  - GitPython-3.1.24-GCCcore-11.2.0.eb:
  - GitPython-3.1.31-GCCcore-12.2.0.eb:
  - GitPython-3.1.40-GCCcore-12.3.0.eb:
  - IPython-7.26.0-GCCcore-11.2.0.eb:
  - IPython-8.5.0-GCCcore-11.3.0.eb:
  - IPython-8.14.0-GCCcore-12.2.0.eb:
  - IPython-8.14.0-GCCcore-12.3.0.eb:
  - IPython-8.17.2-GCCcore-13.2.0.eb:
  - LIBSVM-Python-3.30-foss-2022a.eb:
  - meson-python-0.11.0-GCCcore-12.2.0.eb:
  - meson-python-0.13.2-GCCcore-12.3.0.eb:
  - meson-python-0.15.0-GCCcore-12.3.0.eb:
  - meson-python-0.15.0-GCCcore-13.2.0.eb:
  - mpi4py-3.1.4-gompi-2022b.eb:
  - mpi4py-3.1.4-gompi-2023a.eb:
  - mpi4py-3.1.5-gompi-2023b.eb:
  - netcdf4-python-1.6.1-foss-2022a.eb:
  - netcdf4-python-1.6.3-foss-2022b.eb:
  - netcdf4-python-1.6.4-foss-2023a.eb:
  - netcdf4-python-1.6.5-foss-2023b.eb:
  - PartitionFinder-2.1.1-foss-2020b-Python-2.7.18.eb:
  - pkgconfig-1.5.5-GCCcore-11.3.0-python.eb:
  - pkgconfig-1.5.5-GCCcore-12.2.0-python.eb:
  - pkgconfig-1.5.5-GCCcore-12.3.0-python.eb:
#  - protobuf-python-3.17.3-GCCcore-11.2.0.eb:
  - protobuf-python-3.19.4-GCCcore-11.3.0.eb:
#  - protobuf-python-4.21.9-GCCcore-11.3.0.eb:
  - protobuf-python-4.23.0-GCCcore-12.2.0.eb:
  - protobuf-python-4.24.0-GCCcore-12.3.0.eb:
  - protobuf-python-4.25.3-GCCcore-13.2.0.eb:
#  - PyTables-3.5.2-foss-2020b-Python-2.7.18.eb:
  - Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb:
  - Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb:
  - python-casacore-3.5.2-foss-2023b.eb:
  - python-igraph-0.10.6-foss-2022b.eb:
#  - python-isal-1.1.0-GCCcore-11.3.0.eb:
  - python-libsbml-5.20.2-foss-2023b.eb:
#  - python-louvain-0.15-foss-2021b.eb:
  - python-xxhash-3.4.1-GCCcore-12.3.0.eb:
#  - scikit-learn-0.20.4-foss-2020b-Python-2.7.18.eb:
#  - SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb:
  - SciPy-bundle-2022.05-foss-2022a.eb:
  - SciPy-bundle-2023.02-gfbf-2022b.eb:
  - SciPy-bundle-2023.07-gfbf-2023a.eb:
  - SciPy-bundle-2023.11-gfbf-2023b.eb:
  - spglib-python-2.0.0-foss-2022a.eb:
  - spglib-python-2.0.2-gfbf-2022b.eb:
  - wxPython-4.2.1-foss-2022a.eb:
  - Z3-4.10.2-GCCcore-11.3.0-Python-3.10.4.eb:
  - Z3-4.12.2-GCCcore-12.2.0-Python-3.10.8.eb:
#  - Z3-4.12.2-GCCcore-12.3.0-Python-3.11.3.eb:

Then build with:

MODULEPATH=/scratch-nvme/1/casparl/generic/modfiles_no_LD_LIBRARY_PATH/ eb_github_nvme --easystack python_ctypes_easystack.yml --installpath-modules /scratch-nvme/1/casparl/generic/modfiles_no_LD_LIBRARY_PATH/ --installpath-software /scratch-nvme/1/casparl/generic/software_eb_block_pr3352 --include-easyblocks-from-pr 3352 --filter-env-vars=LD_LIBRARY_PATH --rebuild --robot --upload-test-report

(modify the installpath-*'s as desired, likewise for the MODULEPATH).

I've done similar manual tests as @ocaisa in #3352 (comment) , see for details see https://gitlab.com/eessi/support/-/issues/77, specifically how it picks up on the correct libsndfile.so, see https://gitlab.com/eessi/support/-/issues/77#note_2014141725.

I'm happy with this PR as it is. @ocaisa let me know if you are too, then I'll merge it (we both worked on it, but if we both agree with the current form, it's not breaking the 4-eyes principle :)).

ocaisa · 2024-07-30T07:07:45Z

@casparvl I think this is good to go, successful test with EESSI/software-layer#655

casparvl

Lgtm!

Patch ctypes for Python installations when filtering LD_LIBRARY_PATH

32b7445

This was referenced Jun 7, 2024

Corner cases that fail for lack of LD_LIBRARY_PATH EESSI/software-layer#192

Open

Run test step for Python installations easybuilders/easybuild-easyconfigs#20769

Closed

Add test step by default for recent Python installations

f8c2321

ocaisa changed the title ~~Patch ctypes for Python installations when filtering LD_LIBRARY_PATH~~ Patch ctypes for Python installations when filtering LD_LIBRARY_PATH and add testing by default for recent versions Jun 7, 2024

Also skip a curses test

a771002

Add comment on why we skip some tests

9af5750

Class is inherited so restrict application of new test step

ac1ce4e

ocaisa commented Jun 7, 2024

View reviewed changes

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

ocaisa commented Jun 8, 2024

View reviewed changes

easybuild/easyblocks/p/python.py Show resolved Hide resolved

Adjust approach

d07f920

ocaisa commented Jun 10, 2024

View reviewed changes

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

ocaisa changed the title ~~Patch ctypes for Python installations when filtering LD_LIBRARY_PATH and add testing by default for recent versions~~ Patch ctypes for Python installations when filtering LD_LIBRARY_PATH Jun 10, 2024

Split off the testing step to another PR

0cc765f

boegel added bug fix EESSI Related to EESSI project labels Jun 19, 2024

Update python.py

1ee17c0

casparvl mentioned this pull request Jul 30, 2024

Rebuild Python with ctypes fix + add librosa 0.10.1 EESSI/software-layer#655

Open

casparvl approved these changes Jul 30, 2024

View reviewed changes

casparvl merged commit bb4cb6f into easybuilders:develop Jul 30, 2024
41 checks passed

ocaisa deleted the ctypes_python branch July 30, 2024 14:17

boegel changed the title ~~Patch ctypes for Python installations when filtering LD_LIBRARY_PATH~~ Patch ctypes for Python installations when filtering LD_LIBRARY_PATH + fix path to ldconfig when using alternate sysroot Jul 31, 2024

boegel modified the milestones: 4.x, release after 4.9.2 Jul 31, 2024

Patch ctypes for Python installations when filtering LD_LIBRARY_PATH + fix path to ldconfig when using alternate sysroot #3352

Patch ctypes for Python installations when filtering LD_LIBRARY_PATH + fix path to ldconfig when using alternate sysroot #3352

Conversation

ocaisa commented Jun 7, 2024

ocaisa commented Jun 7, 2024 • edited Loading

ocaisa commented Jun 7, 2024 • edited Loading

boegelbot commented Jun 7, 2024

ocaisa commented Jun 7, 2024

boegelbot commented Jun 7, 2024

boegelbot commented Jun 7, 2024

Overview of tested easyconfigs (in order)

ocaisa commented Jun 7, 2024

boegelbot commented Jun 7, 2024

boegelbot commented Jun 7, 2024

Overview of tested easyconfigs (in order)

ocaisa commented Jun 7, 2024

boegelbot commented Jun 7, 2024

boegelbot commented Jun 7, 2024

Overview of tested easyconfigs (in order)

boegelbot commented Jun 7, 2024

Overview of tested easyconfigs (in order)

ocaisa commented Jun 10, 2024

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

ocaisa commented Jul 26, 2024

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

Overview of tested easyconfigs (in order)

casparvl commented Jul 26, 2024

ocaisa commented Jul 30, 2024 • edited Loading

casparvl left a comment

Choose a reason for hiding this comment

Patch `ctypes` for Python installations when filtering `LD_LIBRARY_PATH` + fix path to `ldconfig` when using alternate sysroot #3352

Patch `ctypes` for Python installations when filtering `LD_LIBRARY_PATH` + fix path to `ldconfig` when using alternate sysroot #3352

ocaisa commented Jun 7, 2024 •

edited

Loading

ocaisa commented Jun 7, 2024 •

edited

Loading

ocaisa commented Jul 30, 2024 •

edited

Loading