{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 #12964

boegel · 2021-05-26T14:06:16Z

(created using eb --new-pr)

~~WIP because a handful of tests are still failing...~~ (failing tests fixed via easybuilders/easybuild-framework#3866 + easybuilders/easybuild-easyblocks#2606)

boegel · 2021-10-20T14:08:14Z

@boegelbot please test @ generoso

boegelbot · 2021-10-20T14:10:12Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=12964 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_12964 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 7116

Test results coming soon (I hope)...

- notification for comment with ID 947703906 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2021-10-20T14:19:09Z

Test report by @boegelbot
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
cnx2 - Linux rocky linux 8.4, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/82c4f067684b27c276262764e6df7a18 for a full test report.

akesandgren · 2021-10-21T09:25:49Z

Test report by @akesandgren
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
b-an02.hpc2n.umu.se - Linux Ubuntu 20.04, x86_64, Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Python 3.8.10
See https://gist.github.com/6c30d56d5ecb877aa24f0811e6c61868 for a full test report.

akesandgren · 2021-10-21T13:42:06Z

Looking at the initial part of the numpy build, it looks like it fails to find a BLAS lib completely....

  libraries -Wl,-Bstatic,-Wl,--start-group,mkl_intel_lp64,mkl_intel_thread,mkl_core,-Wl,--end-group,-Wl,-Bdynamic,iomp5 not found in ['/hpc2n/e
b/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0', '/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/include', '/hpc2n/eb/software/im
kl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib', '/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64']
  NOT AVAILABLE

...
lapack_opt_info:
lapack_mkl_info:
  libraries -Wl,-Bstatic,-Wl,--start-group,mkl_intel_lp64,mkl_intel_thread,mkl_core,-Wl,--end-group,-Wl,-Bdynamic,iomp5 not found in ['/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0', '/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/include', '/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib', '/hpc2n/eb/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64']
  NOT AVAILABLE

We might need to patch the numpy search code...

akesandgren · 2021-10-21T13:53:43Z

The numpy from SciPy-bundle 2020.11 have this instead:

  FOUND:
    libraries = ['-Wl,-Bstatic', '-Wl,--start-group', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', '-Wl,--end-group', '-Wl,-Bdynamic', 'iomp5', 'pthread']
    library_dirs = ['/hpc2n/eb/software/imkl/2020.4.304-iimpi-2020b/mkl/lib/intel64', '/hpc2n/eb/software/iccifort/2020.4.304/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/hpc2n/eb/software/imkl/2020.4.304-iimpi-2020b/mkl', '/hpc2n/eb/software/imkl/2020.4.304-iimpi-2020b/mkl/include', '/hpc2n/eb/software/imkl/2020.4.304-iimpi-2020b/mkl/lib']

akesandgren · 2021-10-21T14:42:55Z

The problem may be that intel/2021a does not add /hpc2n/eb/software/intel-compilers/2021.2.0/compiler/2021.2.0/linux/compiler/lib/intel64_lin to LDFLAGS!!!

So numpy cant find iomp5

branfosj · 2021-10-21T15:28:44Z

If I change https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/toolchains/compiler/intel_compilers.py#L53 to be:

        libpaths = [
            'lib',
            os.path.join('compiler', '2021.2.0', 'linux', 'lib', 'x64'),
            os.path.join('compiler', '2021.2.0', 'linux', 'compiler', 'lib', 'intel64_lin'),
        ]

then I get:

  FOUND:
    libraries = ['-Wl,-Bstatic', '-Wl,--start-group', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', '-Wl,--end-group', '-Wl,-Bdynamic', 'iomp5', 'pthread']
    library_dirs = ['/rds/bear-apps/devel/eb-sjb-up/EL8/EL8-cas/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64', '/rds/bear-apps/devel/eb-sjb-up/EL8/EL8-cas/software/intel-compilers/2021.2.0/compiler/2021.2.0/linux/compiler/lib/intel64_lin']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/rds/bear-apps/devel/eb-sjb-up/EL8/EL8-cas/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0', '/rds/bear-apps/devel/eb-sjb-up/EL8/EL8-cas/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/include', '/rds/bear-apps/devel/eb-sjb-up/EL8/EL8-cas/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib']

Without that change I get similar output to #12964 (comment)

The produced Fortran code is incorrect.

akesandgren · 2021-10-21T18:04:12Z

We need easybuilders/easybuild-framework#3866 to get this moving forward.

Disable TestF90Callback since it fails with Intel.

akesandgren · 2021-10-22T16:33:11Z

We need easybuilders/easybuild-easyblocks#2606 too to get this to pass all tests.

akesandgren · 2021-10-22T17:25:14Z

Test report by @akesandgren
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2606
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
b-an02.hpc2n.umu.se - Linux Ubuntu 20.04, x86_64, Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Python 3.8.10
See https://gist.github.com/2c23004e84a5c918fdccfce99cbb7866 for a full test report.

branfosj · 2021-10-23T09:24:44Z

Test report by @branfosj
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2606
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0211u08b.bear.cluster - Linux RHEL 8.3, x86_64, Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (cascadelake), Python 3.6.8
See https://gist.github.com/a35cc11152d71f04b51cc0db0a3d337a for a full test report.

akesandgren · 2021-10-25T08:45:37Z

@boegel Please add numpy-1.20.3_fix-target-test-ccompiler-opt.patch
It fixes a problem that shows up in my production setup but not in the test setup.

branfosj · 2021-10-25T11:10:19Z

@boegel Please add numpy-1.20.3_fix-target-test-ccompiler-opt.patch It fixes a problem that shows up in my production setup but not in the test setup.

In SciPy-bundle-2021.05-foss-2021a.eb the patches are:

numpy-1.20.3_skip-ppc-long-complex-test.patch - PPC, so not needed here
numpy-1.20.3_xfail-test-nan.patch - OpenBLAS related, so not needed here
numpy-1.20.3_fix-target-test-ccompiler-opt.patch - to add

boegel · 2021-10-26T07:50:35Z

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2606
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node2636.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/9ee75ba1c1cc89b1a639544d37551ab2 for a full test report.

…:boegel/easybuild-easyconfigs into 20210526160607_new_pr_SciPy-bundle202105

boegel · 2021-10-26T07:56:32Z

@boegelbot please test @ generoso

boegelbot · 2021-10-26T08:00:17Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=12964 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_12964 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 7146

Test results coming soon (I hope)...

- notification for comment with ID 951658584 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2021-10-26T08:53:39Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cnx1 - Linux rocky linux 8.4, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/f014f0f436331c63b2c18f6a08dda60a for a full test report.

SebastianAchilles · 2021-10-26T09:10:49Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
rocky8-eb - Linux rocky linux 8.4, x86_64, Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (broadwell), Python 3.6.8
See https://gist.github.com/185c3988b3c3f7d44085689ff234148f for a full test report.

branfosj · 2021-10-26T09:41:56Z

Test report by @branfosj
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0211u08b.bear.cluster - Linux RHEL 8.3, x86_64, Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (cascadelake), Python 3.6.8
See https://gist.github.com/a4eb4e5793b871292764aa75d42b2d21 for a full test report.

branfosj · 2021-10-26T10:32:12Z

Test report by @branfosj
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0104u03a.bear.cluster - Linux RHEL 8.4, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/c5e984843f062b672119bb5caafa91f2 for a full test report.

boegel · 2021-10-26T13:00:43Z

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2606
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3103.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/9e6b3f2197546fc2b09f1b0928690fb0 for a full test report.

branfosj

lgtm

branfosj · 2021-10-26T14:10:01Z

Going in, thanks @boegel!

{lang}[intel/2021.03] SciPy-bundle v2021.05 w/ Python 3.9.5 (WIP)

96d4198

boegel added the update label May 26, 2021

boegel marked this pull request as draft May 26, 2021 14:08

boegel added this to the 4.4.0 milestone May 26, 2021

boegel added this to In progress in next EasyBuild release (v4.4.0) via automation May 26, 2021

boegel modified the milestones: 4.4.0, release after 4.4.0 May 29, 2021

update to intel/2021a

6b864d0

boegel changed the title ~~{lang}[intel/2021.03] SciPy-bundle v2021.05 w/ Python 3.9.5 (WIP)~~ {lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 (WIP) Jun 9, 2021

boegel modified the milestones: next release (4.4.1), release after 4.4.1 Jul 2, 2021

boegel modified the milestones: 4.4.2, release after 4.4.2 Sep 2, 2021

Disable TestF90Callback since it fails with Intel.

c6f73f2

The produced Fortran code is incorrect.

Merge pull request #85 from akesandgren/disable_failing_numpy_test

70cff9c

Disable TestF90Callback since it fails with Intel.

Merge branch 'develop' into 20210526160607_new_pr_SciPy-bundle202105

3dacf2a

add extra numpy patch to SciPy-bundle 2021.05 to fix problem with tests

da542bd

boegel mentioned this pull request Oct 26, 2021

fix support for recent imkl version in numexpr easyblock easybuilders/easybuild-easyblocks#2606

Merged

boegel marked this pull request as ready for review October 26, 2021 07:53

boegel changed the title ~~{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 (WIP)~~ {lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 Oct 26, 2021

Merge branch '20210526160607_new_pr_SciPy-bundle202105' of github.com…

1e36195

…:boegel/easybuild-easyconfigs into 20210526160607_new_pr_SciPy-bundle202105

branfosj approved these changes Oct 26, 2021

View reviewed changes

branfosj merged commit e195e4b into easybuilders:develop Oct 26, 2021

boegel deleted the 20210526160607_new_pr_SciPy-bundle202105 branch October 26, 2021 14:11

boegel mentioned this pull request Oct 30, 2021

failing numpy tests on AMD Rome with SciPy-bundle 2021.05 using intel/2021a #14258

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 #12964

{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 #12964

boegel commented May 26, 2021 •

edited

boegel commented Oct 20, 2021

boegelbot commented Oct 20, 2021

boegelbot commented Oct 20, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

branfosj commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 22, 2021

akesandgren commented Oct 22, 2021

branfosj commented Oct 23, 2021

akesandgren commented Oct 25, 2021

branfosj commented Oct 25, 2021

boegel commented Oct 26, 2021

boegel commented Oct 26, 2021

boegelbot commented Oct 26, 2021

boegelbot commented Oct 26, 2021

SebastianAchilles commented Oct 26, 2021

branfosj commented Oct 26, 2021

branfosj commented Oct 26, 2021

boegel commented Oct 26, 2021

branfosj left a comment

branfosj commented Oct 26, 2021

{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 #12964

{lang}[intel/2021a] SciPy-bundle v2021.05 w/ Python 3.9.5 #12964

Conversation

boegel commented May 26, 2021 • edited

boegel commented Oct 20, 2021

boegelbot commented Oct 20, 2021

boegelbot commented Oct 20, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 21, 2021

branfosj commented Oct 21, 2021

akesandgren commented Oct 21, 2021

akesandgren commented Oct 22, 2021

akesandgren commented Oct 22, 2021

branfosj commented Oct 23, 2021

akesandgren commented Oct 25, 2021

branfosj commented Oct 25, 2021

boegel commented Oct 26, 2021

boegel commented Oct 26, 2021

boegelbot commented Oct 26, 2021

boegelbot commented Oct 26, 2021

SebastianAchilles commented Oct 26, 2021

branfosj commented Oct 26, 2021

branfosj commented Oct 26, 2021

boegel commented Oct 26, 2021

branfosj left a comment

Choose a reason for hiding this comment

branfosj commented Oct 26, 2021

boegel commented May 26, 2021 •

edited