Misalignment of packets upon multiple OOB send/recv packets

[rhc54]

Thank you for taking the time to submit an issue!

Background information

@hjelmn Reported that large OOB messages are getting garbled when the OS buffer is smaller than the message size, thus requiring multiple iterations to send/recv the message

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

master, and probably 3.x

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone

Please describe the system on which you are running

• Operating system/version:
• Computer hardware:
• Network type:

---

Details of the problem

@hjelmn can provide. Our default configuration is to set the OOB/TCP send/recv buffers to 128k. In his case, changing that to 4M resolved the problem.

@ggouaillardet Can you please take a look?

[ggouaillardet]

is there a simple way to have large oob messages generated ?

[rhc54]

I think you can start with the orte/test/system/oob_stress.c program and modify it. IIRC, we do still allow the application procs to communicate via OOB, although they no longer need to do so.

Alternatively, you can have the program register itself as some kind of proc that the OOB will recognize.

[ggouaillardet]

from send_msg

            if (NULL != msg->data) {
                /* technically, this should never happen as iov_count
                 * would be 1 for a zero-byte message, and so we cannot
                 * have a case where we write the header and part of the
                 * msg. However, code checkers don't know that and are
                 * fooled by our earlier check for NULL, and so
                 * we silence their warnings by using this check */
                msg->sdptr = (char *)msg->data + rc;
            }

that does not look symmetric with the beginning of the function, and the comment looks wrong too
(but i should be sleeping, so i might be the one being wrong here)
i just read the code and did not test yet, but what if we simply replace this with

            msg->sdptr = (char *)iov[1].iov_base + rc;

[ggouaillardet]

@rhc54 oob_stress does not compile (some internals were changed and the test was not updated accordingly).
that being said, i was simply able to reproduce the issue with

mpirun --host remote_node --mca oob_tcp_sndbuf 65536 true

and #3423 fixes the issue
i will merge if after CI completes

[rhc54]

Thank you!!

[ggouaillardet]

fwiw, the large message is caused by OPAL_HWLOC_TOPO, which is basically the XML representation of the topology. this is ~30k on a xeon node, but ~200k on a KNL node.
compression would work great here ( ~6k with bzip2 --best) and assuming we can agree on a policy to ensure that both ends support the same encryption, that is something i can implement
that could be done in OPAL_HWLOC_TOPO, or in OPAL_STRING (maybe for strings larger than a MCA param) or in oob/tcp and maybe even in oob

[rhc54]

It's already there - it gets done in the modex if the message is above a given size (looks like the limit is a measly 4kbytes right now, which seems too small - I made the size in PMIx much bigger). See https://github.com/open-mpi/ompi/blob/master/orte/mca/grpcomm/base/grpcomm_base_stubs.c#L501
and
https://github.com/open-mpi/ompi/blob/master/orte/util/compress.h#L35

[ggouaillardet]

@rhc54 thanks for the pointer !
i integrated this into #3424 (there is an other place i will use the same method)