{data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension #19758

lexming · 2024-01-31T10:37:44Z

(created using eb --new-pr)

New easyconfig for DeltaLake + update Arrow to properly install pyarrow as a PythonPackage extension.

Depends on:

update Cargo easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths easybuild-easyblocks#3118
move setuptools_scm extension from hatchling to Python easyconfig #19777

…Lake-0.15.1_remove-obsolete-pyarrow-hotfix.patch

lexming · 2024-01-31T14:06:15Z

@boegelbot: please test @ generoso

boegelbot · 2024-01-31T14:10:08Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=19758 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12783

Test results coming soon (I hope)...

- notification for comment with ID 1919172297 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-01-31T14:31:42Z

Test report by @boegelbot
FAILED
Build succeeded for 1 out of 2 (2 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/a62aa2ad812f9de714cf8b192326ecbc for a full test report.

lexming · 2024-01-31T14:41:11Z

@boegelbot: please test @ generoso
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-01-31T14:45:09Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12786

Test results coming soon (I hope)...

- notification for comment with ID 1919240616 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-01-31T15:23:00Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/dd1f123bc4693b8c05376b76e155aed7 for a full test report.

lexming · 2024-02-01T09:46:12Z

@boegelbot: please test @ generoso
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-02-01T09:50:10Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12798

Test results coming soon (I hope)...

- notification for comment with ID 1920919650 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-01T10:28:44Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/1d2ec7bbea167107793370d6560b887e for a full test report.

boegel · 2024-02-01T11:47:50Z

@boegelbot: please test @ jsc-zen3
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-02-01T11:50:09Z

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3535

Test results coming soon (I hope)...

- notification for comment with ID 1921145037 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-01T12:02:54Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
FAILED
Build succeeded for 1 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/5269f482ffbe5687afd89e05cb685689 for a full test report.

boegel · 2024-02-01T12:03:57Z

@lexming Hidding Perl module dependency?

Can't locate IPC/Cmd.pm in @INC

lexming · 2024-02-01T14:34:27Z

@boegelbot: please test @ jsc-zen3
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-02-01T14:40:08Z

@lexming: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3543

Test results coming soon (I hope)...

- notification for comment with ID 1921476148 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-01T14:57:41Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/58fcafe664b3c19f359907551edc6c48 for a full test report.

lexming · 2024-02-01T15:00:45Z

@boegel very much, fixed 👍

easybuild/easyconfigs/d/DeltaLake/DeltaLake-0.15.1-gfbf-2023a.eb

boegel · 2024-02-01T17:32:07Z

@lexming The changes you made to Arrow are not working for me (see upcoming test report), why are those required?

It would be better to open a dedicated PR for this if there's bug fix needed there...

boegel · 2024-02-01T17:49:54Z

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
FAILED
Build succeeded for 0 out of 2 (2 easyconfigs in total)
node3161.skitty.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/boegel/88003056c18e08d0a8cb86e71d6591fd for a full test report.

branfosj · 2024-02-01T19:31:50Z

@lexming The changes you made to Arrow are not working for me (see upcoming test report), why are those required?

If I've followed this correctly:

DeltaLake requires pyarrow>=8
The changes to Arrow-14.0.1-gfbf-2023a.eb mean that it correctly installs as 14.0.1 instead of 0.0.0.
The changes to the Arrow easyconfig require move setuptools_scm from hatchling to Python easyconfig #19651 to make setuptools_scm available.
19651 targeted 5.0.x and this PR targets develop

This means that merging the changes to Arrow will cause it to be broken in develop until we collapse in the 5.0.x branch.

lexming · 2024-02-01T22:30:53Z

@branfosj that's correct, changes to Arrow in this PR make it to properly install as a PythonPackage extension which triggers extra checks such as pip check. This will capture the wrong 0.0.0 version unless a Python with setuptools_scm is used.

I forgot that we put the changes on setuptools_scm into 5.0 (I have it deployed in my dev system already 😋 ). Maybe we should move it to develop to avoid perpetuating such issues for too long.

@boegel let me know what target branch do you prefer and I'll update this PR accordingly.

lexming · 2024-02-02T08:56:10Z

@branfosj @boegel PR to merge the changes for Python open: #19777

…asyconfigs into 20240131113739_new_pr_DeltaLake0151

lexming · 2024-02-02T17:23:15Z

@boegelbot: please test @ generoso

boegelbot · 2024-02-02T17:25:12Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=19758 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12827

Test results coming soon (I hope)...

- notification for comment with ID 1924341669 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-02T17:45:09Z

Test report by @boegelbot
FAILED
Build succeeded for 1 out of 2 (2 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/b263e1a0d1a9127fd54515e394586e8b for a full test report.

lexming · 2024-02-02T18:08:01Z

@boegelbot: please test @ generoso
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-02-02T18:10:05Z

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12828

Test results coming soon (I hope)...

- notification for comment with ID 1924427666 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-02T18:48:27Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/37145f007e0bc3c8b9416d45ed4ef895 for a full test report.

lexming · 2024-02-02T23:00:02Z

@boegelbot: please test @ jsc-zen3
EB_ARGS=" --include-easyblocks-from-pr=3118"

boegelbot · 2024-02-02T23:00:10Z

@lexming: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=19758 EB_ARGS=" --include-easyblocks-from-pr=3118" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_19758 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3555

Test results coming soon (I hope)...

- notification for comment with ID 1924877387 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-02T23:17:35Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/e1fed8cfb1cd234a60b1f6811fded258 for a full test report.

lexming · 2024-02-03T11:21:37Z

@boegel Python fixed in develop, tests from the bots are fine. Please try again.

Micket · 2024-02-07T17:13:32Z

Test report by @Micket
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3118
SUCCESS
Build succeeded for 3 out of 3 (2 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/b22b34c9af98456b35166f15d02361d0 for a full test report.

fixed

Micket

lgtm

Micket · 2024-02-09T16:18:06Z

Going in, thanks @lexming!

adding easyconfigs: DeltaLake-0.15.1-gfbf-2023a.eb and patches: Delta…

286d341

…Lake-0.15.1_remove-obsolete-pyarrow-hotfix.patch

lexming added the new label Jan 31, 2024

use extension to install pyarrow in Arrow-14.0.1-gfbf-2023a.eb

2ac7cf4

lexming mentioned this pull request Jan 31, 2024

update Cargo easyblock to extract crates into a vendor subdir and overwrite git repo URLs with local paths easybuilders/easybuild-easyblocks#3118

Merged

lexming mentioned this pull request Feb 1, 2024

{data}[gfbf/2023a] polars v0.19.19 #19770

Open

4 tasks

add Perl as build dependency of DeltaLake-0.15.1-gfbf-2023a.eb

1245f43

boegel previously requested changes Feb 1, 2024

View reviewed changes

easybuild/easyconfigs/d/DeltaLake/DeltaLake-0.15.1-gfbf-2023a.eb Outdated Show resolved Hide resolved

remove trailing whitespace in DeltaLake easyconfig

88380e5

easybuilders deleted a comment from boegelbot Feb 1, 2024

lexming changed the title ~~{data}[gfbf/2023a] DeltaLake v0.15.1~~ {data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension Feb 1, 2024

lexming added the bug fix label Feb 1, 2024

Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…

fb9b6ba

…asyconfigs into 20240131113739_new_pr_DeltaLake0151

Micket added this to the release after 4.9.0 milestone Feb 7, 2024

Micket approved these changes Feb 9, 2024

View reviewed changes

Micket merged commit 011eded into easybuilders:develop Feb 9, 2024
9 checks passed

lexming deleted the 20240131113739_new_pr_DeltaLake0151 branch February 9, 2024 21:52

jfgrimm mentioned this pull request Feb 20, 2024

Arrow-14.0.1-gfbf-2023a build fails: python package shows version 0.0.0 #19849

Closed

jfgrimm mentioned this pull request Mar 8, 2024

Add poetry to builddeps to fix pyarrow version #20072

Closed

casparvl mentioned this pull request Mar 21, 2024

{data}[gfbf/2023a] HF-Datasets v2.18.0 #20166

Merged

{data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension #19758

{data}[gfbf/2023a] DeltaLake v0.15.1 and fix Arrow v14.0.1 to install pyarrow as an extension #19758

Conversation

lexming commented Jan 31, 2024 • edited by Micket

lexming commented Jan 31, 2024

boegelbot commented Jan 31, 2024

boegelbot commented Jan 31, 2024

lexming commented Jan 31, 2024

boegelbot commented Jan 31, 2024

boegelbot commented Jan 31, 2024

lexming commented Feb 1, 2024

boegelbot commented Feb 1, 2024

boegelbot commented Feb 1, 2024

boegel commented Feb 1, 2024

boegelbot commented Feb 1, 2024

boegelbot commented Feb 1, 2024

boegel commented Feb 1, 2024

lexming commented Feb 1, 2024

boegelbot commented Feb 1, 2024

boegelbot commented Feb 1, 2024

lexming commented Feb 1, 2024

boegel commented Feb 1, 2024

boegel commented Feb 1, 2024

branfosj commented Feb 1, 2024

lexming commented Feb 1, 2024 • edited

lexming commented Feb 2, 2024

lexming commented Feb 2, 2024

boegelbot commented Feb 2, 2024

boegelbot commented Feb 2, 2024

lexming commented Feb 2, 2024

boegelbot commented Feb 2, 2024

boegelbot commented Feb 2, 2024

lexming commented Feb 2, 2024

boegelbot commented Feb 2, 2024

boegelbot commented Feb 2, 2024

lexming commented Feb 3, 2024

Micket commented Feb 7, 2024

Micket left a comment

Choose a reason for hiding this comment

Micket commented Feb 9, 2024

lexming commented Jan 31, 2024 •

edited by Micket

lexming commented Feb 1, 2024 •

edited