{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.11.0, PMIx 3.1.5 #11333

boegel · 2020-09-23T09:36:28Z

(created using eb --new-pr)
requires ~~easybuilders/easybuild-easyblocks#2184~~ + ~~#11320~~ (UCX) + ~~#11332~~ (hwloc)

…11.0, PMIx 3.1.5

boegel · 2020-09-23T11:10:44Z

@boegelbot please test @ generoso

boegelbot · 2020-09-23T11:15:13Z

@boegel: Request for testing this PR well received on generoso

PR test command 'EB_PR=11333 EB_ARGS= /apps/slurm/default/bin/sbatch --job-name test_PR_11333 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 7856

Test results coming soon (I hope)...

- notification for comment with ID 697295413 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2020-09-23T11:28:36Z

Test report by @boegelbot
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
generoso-x-1 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/b62bfc33c94b6a4fd0d17ee00f8f1e75 for a full test report.

boegel · 2020-09-23T11:42:41Z

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node3406.kirlia.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 2.7.5
See https://gist.github.com/ba213ae6dbf1b2f33bf35108c19d1e84 for a full test report.

boegel · 2020-09-23T11:43:50Z

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node3149.skitty.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/b1a7fec7331b23b529b5f01a34bb735b for a full test report.

boegel · 2020-09-23T14:02:13Z

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node2609.swalot.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 2.7.5
See https://gist.github.com/54e0c16b41bdeb8e18cf5527a0ea4afd for a full test report.

…asyconfigs into 20200923113619_new_pr_OpenMPI405

lexming · 2020-09-25T09:04:35Z

Test report by @lexming
SUCCESS
Build succeeded for 5 out of 5 (4 easyconfigs in this PR)
node127.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, Python 2.7.5
See https://gist.github.com/e6d314163c5aedf8d32de7f0f9056d9f for a full test report.

lexming · 2020-09-25T09:06:50Z

Test report by @lexming
SUCCESS
Build succeeded for 5 out of 5 (4 easyconfigs in this PR)
node376.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/07f7fa869c7e57849c243ebc26b6175d for a full test report.

lexming

This OpenMPI is not working well on my side. A simple MPI hello world program fails to initialise OpenFabrics

$ mpirun ./test
[node379.hydra.os:24944] [[51950,0],0] ORTE_ERROR_LOG: Out of resource in file util/show_help.c at line 501
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   node378
  Local device: mlx5_0
--------------------------------------------------------------------------
Hello world from processor node379.hydra.os, rank 0 out of 2 processors
Hello world from processor node378.hydra.os, rank 1 out of 2 processors

OSU-Micro-benchmarks has the same issue

# OSU MPI Latency Test v5.6.3
# Size          Latency (us)
1024                    2.08
2048                    2.83
4096                    3.72
8192                    5.46
16384                   7.56
32768                   9.83
65536                  14.34
131072                 22.34
262144                 32.28
524288                 54.46
1048576                97.91
2097152               181.33
4194304               354.37
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   node378
  Local device: mlx5_0
--------------------------------------------------------------------------
[node379.hydra.os:15539] [[38701,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 501

The execution completes in both cases, but those errors are not good.

terjekv · 2020-09-25T20:03:48Z

Started a test build on a "clean" arm box. It'll take a bit. It started building M4... The box has no toolchains. :)

terjekv · 2020-09-25T21:47:18Z

Test report by @terjekv
SUCCESS
Build succeeded for 37 out of 37 (4 easyconfigs in this PR)
arm2 - Linux ubuntu 18.04, AArch64, UNKNOWN, Python 3.6.9
See https://gist.github.com/3f3bd7ad22e365787aba06bd717bc751 for a full test report.

boegel · 2020-09-26T09:20:06Z

This OpenMPI is not working well on my side. A simple MPI hello world program fails to initialise OpenFabrics

The problem here is that we should be configuring OpenMPI with --without-verbs when we're using UCX.
That certainly fixes the problem for me (and it's a known issue, see https://www.open-mpi.org/faq/?category=all#ofa-device-error.

Please try again with the updated OpenMPI easyblock from easybuilders/easybuild-easyblocks#2188 .

lexming · 2020-09-26T10:15:31Z

Test report by @lexming
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in this PR)
node128.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, Python 2.7.5
See https://gist.github.com/96567ab27531f1376f4b0764aa6edc36 for a full test report.

lexming · 2020-09-26T10:17:10Z

Test report by @lexming
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in this PR)
node378.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/6314f78e9edcc6d1e3dfd75fe5257922 for a full test report.

lexming · 2020-09-26T10:22:28Z

@boegel thanks a lot, that was indeed the issue. We have been already disabling verbs in our production system, but I was totally misled by the ORTE_ERROR_LOG: Data unpack would read past... error.

lexming

LGTM

boegel · 2020-09-30T07:01:48Z

@lexming So let's merge? Or do you want to see more tests?

boegel · 2020-09-30T20:28:02Z

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node3502.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7302P 16-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/bb1df99da5e1cdd18ba1f4d58c1b7d14 for a full test report.

lexming · 2020-09-30T20:52:40Z

Going in, thanks @boegel !

{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.…

bc91169

…11.0, PMIx 3.1.5

boegel added the update label Sep 23, 2020

boegel added this to the next release (4.3.1) milestone Sep 23, 2020

boegel added the 2020b issues & PRs related to 2020b label Sep 24, 2020

Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…

40c25ee

…asyconfigs into 20200923113619_new_pr_OpenMPI405

easybuilders deleted a comment from boegelbot Sep 25, 2020

lexming reviewed Sep 25, 2020

View reviewed changes

lexming mentioned this pull request Sep 26, 2020

configure OpenMPI 4.x with --without-verbs when using UCX easybuilders/easybuild-easyblocks#2188

Merged

lexming approved these changes Sep 26, 2020

View reviewed changes

boegel added this to In progress in EasyBuild v4.3.1 via automation Sep 26, 2020

lexming merged commit ef4e18c into easybuilders:develop Sep 30, 2020

EasyBuild v4.3.1 automation moved this from In progress to Done Sep 30, 2020

boegel deleted the 20200923113619_new_pr_OpenMPI405 branch October 2, 2020 20:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.11.0, PMIx 3.1.5 #11333

{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.11.0, PMIx 3.1.5 #11333

boegel commented Sep 23, 2020 •

edited

boegel commented Sep 23, 2020

boegelbot commented Sep 23, 2020

boegelbot commented Sep 23, 2020

boegel commented Sep 23, 2020

boegel commented Sep 23, 2020

boegel commented Sep 23, 2020

lexming commented Sep 25, 2020

lexming commented Sep 25, 2020

lexming left a comment

terjekv commented Sep 25, 2020

terjekv commented Sep 25, 2020

boegel commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming left a comment

boegel commented Sep 30, 2020

boegel commented Sep 30, 2020

lexming commented Sep 30, 2020

{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.11.0, PMIx 3.1.5 #11333

{lib}[GCCcore/10.2.0] OpenMPI v4.0.5, libevent v2.1.12, libfabric v1.11.0, PMIx 3.1.5 #11333

Conversation

boegel commented Sep 23, 2020 • edited

boegel commented Sep 23, 2020

boegelbot commented Sep 23, 2020

boegelbot commented Sep 23, 2020

boegel commented Sep 23, 2020

boegel commented Sep 23, 2020

boegel commented Sep 23, 2020

lexming commented Sep 25, 2020

lexming commented Sep 25, 2020

lexming left a comment

Choose a reason for hiding this comment

terjekv commented Sep 25, 2020

terjekv commented Sep 25, 2020

boegel commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming commented Sep 26, 2020

lexming left a comment

Choose a reason for hiding this comment

boegel commented Sep 30, 2020

boegel commented Sep 30, 2020

lexming commented Sep 30, 2020

boegel commented Sep 23, 2020 •

edited