Processes returns wrong results on OpenMPI 4.0.4 unless $OMPI_MCA_pml is set to "^ucx" #8321

ksiazekm · 2020-12-27T22:54:35Z

OpenMPI 4.0.4
Installed from official Fedora repository with use of dnf.
OS: Fedora 33, the latest
gcc 10.2.1 20201125 (Red Hat 10.2.1-9)
AMD FX8350,

some processes randomly return wrong results on the given configuration. When
tested the same source code on Debian GNU/Linux 7 (wheezy) with OpenMPI 1.4.5
it returns correct results each time. Test on each configuration was executed
30 times in a loop. OpenMPI 4.0.4 started to work properly after set
$OMPI_MCA_pml to "^ucx".

The source code: gist link

hppritcha · 2021-01-04T20:42:18Z

It would be useful to know the version of UCX is being used.

ksiazekm · 2021-01-05T12:48:30Z

It would be useful to know the version of UCX is being used.

ucx.x86_64 1.8.1-3.fc33

jsquyres · 2021-01-05T14:42:24Z

Can you provide more detail about exactly what is going wrong, and/or a small example showing the problem?

jsquyres · 2021-01-05T16:05:00Z

I'm the worst -- you provided a gist link with a sample and I missed it. We'll investigate...

hppritcha · 2021-01-08T22:43:18Z

@ksiazekm i'm having reproducing this. First how many MPI processes are you using when you observe the problem and what is the tabsize you're supplying?

ksiazekm · 2021-01-09T13:09:38Z

Here you have 30 executions of the code from the previous gist: Test Results

ggouaillardet · 2021-01-09T16:12:32Z

@hppritcha FWIW, I was able to reproduce the issue on a Fedora 33 virtual machine, using Open MPI and UCX provided by the distro. I was only able to find discarding the pml/ucx component was a workaround.
I did not have time to dig a bit more and figure out whether the root cause is in Open MPI, or UCX or both.

awlauria · 2021-01-11T15:03:25Z

So far I can't reproduce on power9 + rhel8.2. I tried v4.0.x latest and also the v4.0.4 release tarball.

hoopoepg · 2021-01-12T14:36:47Z

@hppritcha FWIW, I was able to reproduce the issue on a Fedora 33 virtual machine, using Open MPI and UCX provided by the distro

hi
have you any specific configuration in virtual machine? what processor of host system?
we are trying to reproduce issue but have no success

ggouaillardet · 2021-01-12T14:48:30Z

That is a pretty standard Virtualbox VM running on a x86_64 (intel) processor.

hoopoepg · 2021-01-12T14:50:42Z

what is number of CPU? memory volume?
and could you run there ucx_info -d there & post output here?

thank you

ggouaillardet · 2021-01-12T15:07:43Z

1 GB
1 CPU

#
# Memory domain: sockcm
#     Component: sockcm
#           supports client-server connection establishment via sockaddr
#   < no supported devices found >
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#   Transport: tcp
#      Device: enp0s3
#
#      capabilities:
#            bandwidth: 113.16/ppn + 0.00 MB/sec
#              latency: 5776 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to iface
#             priority: 1
#       device address: 4 bytes
#        iface address: 2 bytes
#       error handling: none
#
#
# Connection manager: tcp
#      max_conn_priv: 2040 bytes
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#   Transport: self
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: none
#
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#   Transport: sysv
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: none
#
#
# Memory domain: posix
#     Component: posix
#             allocate: unlimited
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#   Transport: posix
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: none
#

ggouaillardet · 2021-01-12T15:08:34Z

use

mpirun --oversubscribe -np 4 a.out 120000000

you might have to run it several times to evidence the issue.

jsquyres · 2021-01-12T16:10:28Z

I'm pre-emptively adding a v4.1 label on this issue too -- I'm assuming that the issue exists over there as well.

gpaulsen · 2021-01-12T16:12:32Z

I set the blocker label until we have more information.

ggouaillardet · 2021-01-13T00:51:10Z

@jsquyres @gpaulsen I was only able to reproduce this on a Fedora 33 (virtual) machine, and the workaround is to discard the pml/ucx component (built on top of the distro provided UCX).

I did not investigate more (lack of time) and could not reach a conclusion w.r.t. the root cause of this issue (Open MPI? UCX? Fedora? a combination of these?)

@opoplawski meanwhile, you might want to skip the pml/ucx component in the Fedora package by default. That can be achieved by adding a

pml = ^ucx

line into /etc/openmpi-x86_64/openmpi-mca-params.conf

jsquyres · 2021-01-13T14:59:09Z

@jladd-mlnx @hoopoepg Someone from the UCX community needs to look into this.

hoopoepg · 2021-01-13T15:00:18Z

we can't reproduce issue on our environment :(

opoplawski · 2021-01-28T14:07:24Z

I'm happy to push a ucx 1.9 update to Fedora 33 if that seems appropriate. Is updating UCX versions okay? There isn't a soname bump.

hppritcha · 2021-01-29T17:23:15Z

@opoplawski thanks Orion. Transferring to Nvidia to let them decide how to deal with this from the Open MPI side. Recommend a configure check to not build UCX pml and osc if the UCX being installed is older than 1.9. @hoopoepg

jsquyres · 2021-01-29T17:30:58Z

@hppritcha Better to use the general "UCX" team name to alert all the relevant people on the UCX side, not just @hoopoepg.

FYI: @open-mpi/ucx

yosefe · 2021-01-29T17:47:57Z

@hppritcha @jsquyres how about pml/ucx would disqualify itself at runtime, if UCX library version is older than v1.9, with some warning and mca var to override this?
I guess we could update that minimal version from time to time as we move to newer UCX versions

jsquyres · 2021-01-29T18:04:16Z

If you really want to stop supporting older versions of UCX, you can certainly do that.

IMHO: It would be best to have both configure.m4-time and run time checks for versions, just to be safe.

opoplawski · 2021-01-30T17:02:48Z

I've submitted ucx 1.9 to Fedora 33 - https://bodhi.fedoraproject.org/updates/FEDORA-2021-613166cadb Please test and provide feedback.

jsquyres · 2021-02-01T20:11:45Z

@ksiazekm Just out of curiosity, are you seeing the same behavior with Open MPI v4.1.0 as well?

jsquyres · 2021-02-08T20:27:42Z

@open-mpi/ucx Can you give a definitive ruling on what you plan to do? I.e., are UCX versions prior to v1.9.0 problematic?

hoopoepg · 2021-02-09T07:17:44Z

there is one more issue #8442 which may be related to this one, and we reproduced it on our environment. working over idenrifying.

hoopoepg · 2021-02-09T10:21:36Z

@ksiazekm could you test latest OMPI (4.0.5)+UCX (1.9) available in Fedora 33?

thank you

ksiazekm · 2021-02-09T16:46:35Z

I can confirm that it works properly with:

ucx.x86_64 1.9.0-1.fc33
openmpi.x86_64 4.0.5-1.fc33

hppritcha · 2021-02-11T18:49:54Z

@open-mpi/ucx just to note there seem to be some significant differences between this issue and #8442 namely this one doesn't seem to be reproducible when using RC UCX_TLS, whereas #8442 seems to be avoidable using sm, tcp. Still they may be related although the conditions under which they appear wrt UCX_TLS are different.

hoopoepg · 2021-02-15T16:14:06Z

@ksiazekm could you provide output of command ```ucx_info -d" from system where data corruption happened?

thank you

hoopoepg · 2021-02-16T13:57:18Z

@open-mpi/ucx Can you give a definitive ruling on what you plan to do? I.e., are UCX versions prior to v1.9.0 problematic?

Hi @jsquyres
UCX v1.8 has data race in some configuration when used TCP and SHM transports simultaneously. In UCX v1.9 this data flow was updated and issue doesn’t appear anymore. The final fix was implemented in UCX v1.10 in PR openucx/ucx#5936.

Is it still important enough to add warning given that we are pushing updated package for Fedora?

jsquyres · 2021-02-16T15:38:27Z

Many people use Open MPI outside of Fedora.

If you want to restrict Open MPI to only use UCX >= v1.10, you need to update the configury.

Silent data corruption is BAD.

yosefe · 2021-02-16T15:57:55Z

@jsquyres IMO, runtime version check would be more strict (in case the user compiles OpenMPI with one UCX version, but runs with another version), WDYT?

jsquyres · 2021-02-16T16:01:43Z

That would be fine as well.

But we definitely also like configure-time failures (with helpful explanation messages). E.g., if someone tries to compile with an old/unsupported UCX, it should fail right away during configure (vs. succeeding to configure, build, and install, and then only fail at run time).

Make sense?

yosefe · 2021-02-16T16:09:38Z

@jsquyres yes, that makes sense. just to make sure, it would not be a problem for OpenMPI that we essentially drop support for older UCX versions?

yosefe · 2021-02-16T16:10:44Z

adding @shamisp as well

ksiazekm · 2021-02-16T23:18:47Z

@hoopoepg here you have:

$ ucx_info -d
#
# Memory domain: sockcm
#     Component: sockcm
#           supports client-server connection establishment via sockaddr
#   < no supported devices found >
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#   Transport: tcp
#      Device: enp3s6
#
#      capabilities:
#            bandwidth: 11.32/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to iface
#             priority: 1
#       device address: 4 bytes
#        iface address: 2 bytes
#       error handling: none
#
#
# Connection manager: tcp
#      max_conn_priv: 2040 bytes
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#   Transport: self
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: none
#
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#   Transport: sysv
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: none
#
#
# Memory domain: posix
#     Component: posix
#             allocate: unlimited
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#   Transport: posix
#      Device: memory
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#             priority: 0
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: none
#

hoopoepg · 2021-02-17T07:27:52Z

@ksiazekm thank you for info.
You have same configuration as we reproduced issue and upgrade to UCX 1.9 should resolve issue.

ksiazekm · 2021-02-17T16:04:12Z

@hoopoepg indeed, as I wrote in the comment :)

jsquyres · 2021-02-17T16:46:39Z

@jsquyres yes, that makes sense. just to make sure, it would not be a problem for OpenMPI that we essentially drop support for older UCX versions?

Please fix #8489, and then if you strand your customers with older versions of UCX, I don't really care.

jsquyres · 2021-02-22T20:49:28Z

@open-mpi/ucx What version of UCX is known to be good -- is it v1.9.0, or the upcoming v1.10.0? (I ask because I see that 1.9.0 is still the current release on https://github.com/openucx/ucx).

Given that this is silent data corruption, and per the discussion above, it sounds like we need both a configure-time check and a run-time check to ensure that the UCX PML is running with >= UCX v1.GOOD_VERSION. Is there any progress on this? We need this ASAP to get new Open MPI releases out the door.

shamisp · 2021-02-23T00:32:59Z

@jsquyres , FYI we documented this critical issue on the front page https://github.com/openucx/ucx#known-critical-issues

jsquyres added the Target: v4.0.x label Jan 5, 2021

jsquyres added the question label Jan 5, 2021

hppritcha self-assigned this Jan 5, 2021

awlauria mentioned this issue Jan 12, 2021

[from StackOverflow] MPI returns incorrect results for one of processes #8310

Closed

jsquyres added this to the v4.0.6 milestone Jan 12, 2021

jsquyres added the Target: v4.1.x label Jan 12, 2021

gpaulsen added the Severity: blocker label Jan 12, 2021

jsquyres added bug and removed question labels Jan 12, 2021

jsquyres assigned hoopoepg and hppritcha and unassigned hppritcha Jan 13, 2021

hppritcha removed their assignment Jan 29, 2021

jayeshkrishna mentioned this issue Feb 3, 2021

Data corruption with OpenMPI 4.0.4 and UCX 1.9.0 #8442

Closed

yosefe mentioned this issue Feb 22, 2021

ucx: disable version 1.8 #8511

Merged

jsquyres closed this as completed in #8511 Feb 23, 2021

cponder mentioned this issue Mar 17, 2023

How to run on node without infiniband? openucx/ucx#8952

Closed

Processes returns wrong results on OpenMPI 4.0.4 unless $OMPI_MCA_pml is set to "^ucx" #8321

Processes returns wrong results on OpenMPI 4.0.4 unless $OMPI_MCA_pml is set to "^ucx" #8321

Comments

ksiazekm commented Dec 27, 2020

hppritcha commented Jan 4, 2021

ksiazekm commented Jan 5, 2021

jsquyres commented Jan 5, 2021

jsquyres commented Jan 5, 2021

hppritcha commented Jan 8, 2021

ksiazekm commented Jan 9, 2021

ggouaillardet commented Jan 9, 2021

awlauria commented Jan 11, 2021

hoopoepg commented Jan 12, 2021

ggouaillardet commented Jan 12, 2021

hoopoepg commented Jan 12, 2021

ggouaillardet commented Jan 12, 2021

ggouaillardet commented Jan 12, 2021

jsquyres commented Jan 12, 2021

gpaulsen commented Jan 12, 2021

ggouaillardet commented Jan 13, 2021

jsquyres commented Jan 13, 2021

hoopoepg commented Jan 13, 2021

opoplawski commented Jan 28, 2021 • edited Loading

hppritcha commented Jan 29, 2021

jsquyres commented Jan 29, 2021

yosefe commented Jan 29, 2021

jsquyres commented Jan 29, 2021

opoplawski commented Jan 30, 2021

jsquyres commented Feb 1, 2021

jsquyres commented Feb 8, 2021

hoopoepg commented Feb 9, 2021

hoopoepg commented Feb 9, 2021 • edited Loading

ksiazekm commented Feb 9, 2021

hppritcha commented Feb 11, 2021

hoopoepg commented Feb 15, 2021

hoopoepg commented Feb 16, 2021

jsquyres commented Feb 16, 2021

yosefe commented Feb 16, 2021

jsquyres commented Feb 16, 2021

yosefe commented Feb 16, 2021

yosefe commented Feb 16, 2021

ksiazekm commented Feb 16, 2021 • edited Loading

hoopoepg commented Feb 17, 2021

ksiazekm commented Feb 17, 2021 • edited Loading

jsquyres commented Feb 17, 2021

jsquyres commented Feb 22, 2021

shamisp commented Feb 23, 2021

opoplawski commented Jan 28, 2021 •

edited

Loading

hoopoepg commented Feb 9, 2021 •

edited

Loading

ksiazekm commented Feb 16, 2021 •

edited

Loading

ksiazekm commented Feb 17, 2021 •

edited

Loading