IBM dynamic/no-disconnect test hangs

[rhc54]

The IBM dynamic/no-disconnect test is hanging, and then eventually "failing" when the PMIx publish/lookup exchange times out. This only occurs if multiple nodes are involved in the job - if you run the test on a single node, it will pass.

I have run the test on two nodes, using the following command:

mpirun -n 2 --leave-session-attached ./no-disconnect

with a default hostfile containing two nodes, each with 24 slots. After a lot of print statements and perusing output, I have found that:

• the test spawns a total of 32 processes. Thus, the last 8 procs "bleed" onto the second node. This is where the problem occurs.

• the procs on the 2nd node issue a lookup on their "accept" key, but that key is never published. This is why we hang. However, I see all the "connect" keys with the same signature sitting in the orte_data_server queue.

Unfortunately, I don't understand this test sufficiently to know which process is supposed to be publishing the corresponding key, and so I cannot track down why it isn't doing so. I added a lot of prints to the code path, and all data that is published appears to be getting into the orte_data_server as it should. Likewise, lookup is going thru the entire collected data.

@ggouaillardet Would you have some time to dig into this a bit more? I have added a verbosity to the orte_data_server (orte_data_server_verbose) to aid in debugging, plus some additional verbose statements in the orted PMIx server. See PR #3524

[ggouaillardet]

@rhc54 the issue is unlikely in the publish/lookup part ...
i can reproduce the issue with 2 nodes, a total of 8 tasks and the patch below

then, from n0

n0 $ mpirun --host n0:5,n1:3 -np 2 ./no-disconnect | grep "I AM"
[1,0]<stdout>:I AM pid 55780 with level 0
[1,1]<stdout>:I AM pid 55781 with level 0
[2,0]<stdout>:I AM pid 55786 with level 1
[3,0]<stdout>:I AM pid 55787 with level 1
[4,0]<stdout>:I AM pid 55792 with level 2
[5,0]<stdout>:I AM pid 40083 with level 2
[6,0]<stdout>:I AM pid 40086 with level 2
[7,0]<stdout>:I AM pid 55795 with level 2
[7,0]<stdout>:I AM pid 40089 with level 2

<--- HANG -->

for an unkown reason, job [7,0] was spawned twice (!)

fwiw, here is a possible process tree (there is no synchronization, and tasks are spawned in parallel

[0,0] mpirun
  |--- [1,0]
  |      |--- [2,0]
  |             |--- [4,0]
  |             |--- [6,0] 
  |--- [1,1]
  |      |--- [3,0]
  |             |--- [5,0]
  |             |--- [7,0]

note

• MPI_Comm_spawn() is always invoked with count=1
• there is no synchronization, so [7,0] is usually spawned on n0 (instead of n1),
  i do not suspect a bug, but i'd rather suspect a task running on n0 (such ask [4,0]) has already exited and the first available slot is on n0

i hope that is useful and you can resume your investigation from there.

diff --git a/ibm/dynamic/no-disconnect.c b/ibm/dynamic/no-disconnect.c
index 2230761..224fa37 100644
--- a/ibm/dynamic/no-disconnect.c
+++ b/ibm/dynamic/no-disconnect.c
@@ -13,11 +13,13 @@
 #include <stdlib.h>
 #include <stdio.h>
 #include <string.h>
+#include <sys/utsname.h>
+
 #include <mpi.h>
 #include "nice_msgs.h"
 
 #define NCHARS 30
-const int max_depth = 4;
+const int max_depth = 2;
 
 
 int main (int argc, char **argv)
@@ -26,6 +28,7 @@ int main (int argc, char **argv)
     char bufr[2][NCHARS];               /* recv buffers */
     MPI_Comm parent;
     int level = 0, participate = 1;
+    struct utsname buf;
 
     /* If this is prior to OMPI v2.0, return 77/skip */
 #if defined(OPEN_MPI)
@@ -38,6 +41,9 @@ int main (int argc, char **argv)
     }
 #endif
 
+    uname(&buf);
+    printf("I AM pid %d with level %d on %s\n", getpid(), (argc < 2)?0:atoi(argv[1]), buf.nodename);
+
     MPI_Init(&argc, &argv);
     MPI_Comm_get_parent(&parent);
 

fun facts (probably a consequence of the very same bug leading to a hang)
still from n0

n0 $ mpirun -np 2 --host n1:5,n2:3 --tag-output ./no-disconnect 2>&1 | grep "I AM"
[1,0]<stdout>:I AM pid 40312 with level 0
[1,1]<stdout>:I AM pid 40313 with level 0
[2,0]<stdout>:I AM pid 40318 with level 1
[3,0]<stdout>:I AM pid 40319 with level 1
[4,0]<stdout>:I AM pid 40324 with level 2
[5,0]<stdout>:I AM pid 39730 with level 2
[6,0]<stdout>:I AM pid 39733 with level 2
[7,0]<stdout>:I AM pid 39734 with level 2

<-- SUCCESS -->

n0 $ mpirun -np 2 --host n2:5,n1:3 --tag-output ./no-disconnect 2>&1 | grep "I AM"
[1,1]<stdout>:I AM pid 39826 with level 0
[1,0]<stdout>:I AM pid 39825 with level 0
[2,0]<stdout>:I AM pid 39831 with level 1
[3,0]<stdout>:I AM pid 39832 with level 1
[4,0]<stdout>:I AM pid 39837 with level 2
[5,0]<stdout>:I AM pid 40400 with level 2
[6,0]<stdout>:I AM pid 40403 with level 2
[7,0]<stdout>:I AM pid 39840 with level 2
[7,0]<stdout>:I AM pid 40406 with level 2

<-- the job completes, though [7,0] was spawned twice -->

[ggouaillardet]

@rhc54 i edited the patch to print the hostname
bottom line, [7,0] is spawned on both n0 and n1

[ggouaillardet]

@rhc54 i digged a bit more

regarding [7,0]
on n0, orte_rmaps_base_compute_vpids is invoked with nodes={n0,n1} but
on n1, orte_rmaps_base_compute_vpids is invoked with nodes={n1}

so it looks like n0 is aware the slot previously allocated to [4,0] was returned, but n1 is not, hence the de-synchronization and the multiple allocation.

[ggouaillardet]

also, my understanding of spawn() internals is that the HNP is doing an xcast() (to all nodes ?) asking to spawn n tasks. that requires that all orted have a consistent view of what is running on which nodes, so each orted can figure out how many tasks should be fork&exec'ed locally.

did i get it right ?

if yes, then what is the rationale for that ?
what i mean, is what if only the HNP had a view of what is running and where, so it can encode what should be spawned and where in the xcast message ?
that would require less memory on orted without requiring more on mpirun.
obvioulsy, the message to be xcast'ed would be larger, but at this stage, my gut feeling is that this is not a deal breaker.

[rhc54]

Yes, you got it right. Your proposal is exactly what we used to do, but the problem is that the launch message (when running at scale) is quite large and significantly slows down the launch. There are a few possible solutions (in order of difficulty of implementation):

• have the HNP notify the orteds of completion so they can update their view of the overall state
• do the backend mapping on initial launch, but force all comm_spawn jobs to be mapped solely by the HNP (and accept the launch performance penalty for that unusual scenario)
• have the HNP determine the number of procs on each node, and find an efficient regex scheme for communicating that info

It'll be a little while before I can get to it, but I appreciate the detective work!

[rhc54]

FWIW: I tried the first and that didn't work - too many race conditions. I now have a prototype for the 3rd option and am debugging it.

[ggouaillardet]

i see, let me know if you need any help debugging it.
assuming we can build a factorized regex, the 3rd option is the best.

back to the 2nd option, in order to tell orted how many tasks should be spawn'ed locally, what we really need is a scatter instead of a broadcast. depending on the routing protocol between orteds, there is room to optimize that (e.g. each orted split the received message, and forward smaller chunks to its children.
and even if we stick to a broadcast, i suspect zlib can do a pretty good job at compresing it
(not as good as regex with regular patterns, but more bulletproof, regardless how regular the pattern is)

[rhc54]

Okay, FWIW I now have a fix that works. It is based on option 3. I have to update the other mappers (I only prototyped round-robin), but should be able to commit it tomorrow.

[rhc54]

Fixed in master and PR'd to v3.0