hwloc 2.0 segfault in ec2/cfncluster on master

[PeterGottesman]

Background information

This was encountered on an EC2 cluster running in a slurm(16.05.3) allocation.

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

This occurred on master(e79eb85).

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Ompi was installed from a git clone, then tarballed and distributed to all other nodes in a cluster.

Please describe the system on which you are running

• Operating system/version: Amazon Linux
• Computer hardware: EC2 C4.8Xlarge
• Network type:

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

shell$ salloc -N 10
shell$ mpirun hostname
--------------------------------------------------------------------------
An ORTE daemon has unexpectedly failed after launch and before
communicating back to mpirun. This could be caused by a number
of factors, including an inability to create a connection back
to mpirun due to a lack of common network interfaces and/or no
route found between them. Please check network connectivity
(including firewalls and network routing requirements).
--------------------------------------------------------------------------

I have a core file for each orted, backtrace from one below
GDB backtrace:

#0  0x00007f89fbdd11d1 in hwloc__duplicate_object (newtopology=newtopology@entry=0x11d1280, newparent=newparent@entry=0x11d1690, newobj=0x11e7530, newobj@entry=0x0,
    src=src@entry=0x1112900) at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:715
#1  0x00007f89fbdd12fb in hwloc__duplicate_object (newtopology=newtopology@entry=0x11d1280, newparent=newparent@entry=0x0, newobj=0x11d1690, src=src@entry=0x10fb2e0)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:735
#2  0x00007f89fbdd34a6 in opal_hwloc2a_hwloc__topology_dup (newp=newp@entry=0x7fff618ecb10, old=old@entry=0x10fae60, tma=tma@entry=0x7fff618ecb20)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:833
#3  0x00007f89fbdaf65c in opal_hwloc2a_hwloc_shmem_topology_get_length (topology=0x10fae60, lengthp=lengthp@entry=0x7f89f93e7fe8 <shmemsize>, flags=flags@entry=0)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/shmem.c:68
#4  0x00007f89f91e646c in init () at ../../../../../orte/mca/rtc/hwloc/rtc_hwloc.c:104
#5  0x00007f89fc0a120b in orte_rtc_base_select () at ../../../../orte/mca/rtc/base/rtc_base_select.c:74
#6  0x00007f89fc0785d2 in orte_ess_base_orted_setup () at ../../../../orte/mca/ess/base/ess_base_std_orted.c:510
#7  0x00007f89f9dff115 in rte_init () at ../../../../../orte/mca/ess/slurm/ess_slurm_module.c:80
#8  0x00007f89fc0364d9 in orte_init (pargc=0x11e7760, pargc@entry=0x7fff618ede9c, pargv=0x40, pargv@entry=0x7fff618ede90, flags=0, flags@entry=2)
    at ../../orte/runtime/orte_init.c:273
#9  0x00007f89fc0596bd in orte_daemon (argc=argc@entry=19, argv=argv@entry=0x7fff618ee0f8) at ../../orte/orted/orted_main.c:350
#10 0x000000000040078a in main (argc=19, argv=0x7fff618ee0f8) at ../../../../orte/tools/orted/orted.c:60

[rhc54]

@bgoglin Can you make anything out of that backtrace?

[bgoglin]

I guess newobj->logical_index is wrong. "print *newobj" in GDB would help.
Otherwise send the xml output of lstopo on that node and I'll see if I can reproduce.

[bgoglin]

Good candidate for easier debugging: if you have a hwloc gitclone built, do "cd tests/hwloc; make hwloc_topology_dup; ./hwloc_topology_dup", it should crash too.

[PeterGottesman]

Okay, running gdb on a core file I get the following result:

(gdb) bt                                       
#0  0x00007ff1736153b1 in hwloc__duplicate_object (newtopology=newtopology@entry=0xb5c3a0, newparent=newparent@entry=0xb5c7b0, newobj=0xb72650, newobj@entry=0x0, src=src@entry=0xa9da20)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:715                 
#1  0x00007ff1736154db in hwloc__duplicate_object (newtopology=newtopology@entry=0xb5c3a0, newparent=newparent@entry=0x0, newobj=0xb5c7b0, src=src@entry=0xa86400)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:735                 
#2  0x00007ff173617686 in opal_hwloc2a_hwloc__topology_dup (newp=newp@entry=0x7fff8ad23720, old=old@entry=0xa85f80, tma=tma@entry=0x7fff8ad23730)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/topology.c:833                 
#3  0x00007ff1735f383c in opal_hwloc2a_hwloc_shmem_topology_get_length (topology=0xa85f80, lengthp=lengthp@entry=0x7ff170c2bfe8 <shmemsize>, flags=flags@entry=0)
    at ../../../../../../../opal/mca/hwloc/hwloc2a/hwloc/hwloc/shmem.c:68                     
#4  0x00007ff170a2a46c in init () at ../../../../../orte/mca/rtc/hwloc/rtc_hwloc.c:104        
#5  0x00007ff1738e620b in orte_rtc_base_select () at ../../../../orte/mca/rtc/base/rtc_base_select.c:74
#6  0x00007ff1738bd5d2 in orte_ess_base_orted_setup () at ../../../../orte/mca/ess/base/ess_base_std_orted.c:510
#7  0x00007ff171643115 in rte_init () at ../../../../../orte/mca/ess/slurm/ess_slurm_module.c:80
#8  0x00007ff17387b4d9 in orte_init (pargc=0xb72880, pargc@entry=0x7fff8ad24abc, pargv=0x40, pargv@entry=0x7fff8ad24ab0, flags=0, flags@entry=2) at ../../orte/runtime/orte_init.c:273
#9  0x00007ff17389e6bd in orte_daemon (argc=argc@entry=19, argv=argv@entry=0x7fff8ad24d18) at ../../orte/orted/orted_main.c:350
#10 0x000000000040078a in main (argc=19, argv=0x7fff8ad24d18) at ../../../../orte/tools/orted/orted.c:60
(gdb) print *newobj                            
$1 = {type = OPAL_HWLOC2A_hwloc_OBJ_OS_DEVICE, subtype = 0x0, os_index = 4294967295, name = 0xb727a0 "xvda", memory = {total_memory = 0, local_memory = 0, page_types_len = 0, 
    page_types = 0x0}, attr = 0xb72770, depth = 4294967291, logical_index = 1, next_cousin = 0x0, prev_cousin = 0x0, parent = 0x0, sibling_rank = 0, next_sibling = 0x0, 
  prev_sibling = 0x0, arity = 0, children = 0x0, first_child = 0x0, last_child = 0x0, symmetric_subtree = 0, io_arity = 0, io_first_child = 0x0, misc_arity = 0, misc_first_child = 0x0, 
  cpuset = 0x0, complete_cpuset = 0x0, allowed_cpuset = 0x0, nodeset = 0x0, complete_nodeset = 0x0, allowed_nodeset = 0x0, infos = 0xb727c0, infos_count = 3, userdata = 0x0, gp_index = 84}

The xml output of lstopo and the output of hwloc_topology_dup (which didn't actually crash) are attached.
hwloc_info.tar.gz

[bgoglin]

Strange, I can't find this "xvda" device in your XML. Maybe because you ran lstopo from 1.11 ? hwloc 1.11 doesn't report as many block devices as git master (you xvda is likely not a PCI device). That would also explain why hwloc_topology_dup didn't fail.

I assumed I was duplicating things out-of-order but I can't reproduce anyway.

Can you also "print i", "print newtopology->slevels[i]", "print level[newobj->logical_index-1]" ?
I assume the last one will display NULL, it will would explain things.

[PeterGottesman]

All of these have been optimized out according to gdb

(gdb) print i           
$3 = <optimized out>    
(gdb) print newtopology->slevels[i]             
value has been optimized out                    
(gdb) print level[newobj->logical_index-1]      
value has been optimized out 

[PeterGottesman]

lstopo_out.xml.txt

And here is the corrected lstopo output, sorry about that.

[bgoglin]

Thanks for the XML.
Can you try again in gdb after replacing i with 2 and newobj->logical_index with 1?

[PeterGottesman]

(gdb) print newtopology->slevels[2]
$2 = {nbobjs = 2, objs = 0xb5cbf0, first = 0x0, last = 0xb72650}
(gdb) print level[0]
value has been optimized out

[jsquyres]

@bgoglin and @PeterGottesman iterated off-list and @bgoglin says he'll have a fix for tomorrow.

Just to be clear: hwloc v2.0.0 is not on OMPI v3.0.0, so there's no need to hold up v3.0.0 for this issue.

[bgoglin]

This should fix the issue:

--- hwloc-shmem-20170806.1138.gite5274e0.old/hwloc/topology.c   2017-08-07 22:51:47.040900069 +0000
+++ hwloc-shmem-20170806.1138.gite5274e0/hwloc/topology.c       2017-08-08 08:12:21.074988757 +0000
@@ -656,6 +656,7 @@
 {
   struct hwloc_tma *tma = newtopology->tma;
   hwloc_obj_t *level;
+  unsigned level_width;
   size_t len;
   unsigned i;
   hwloc_obj_t child;
@@ -711,6 +711,7 @@
   if ((int) src->depth < 0) {
     i = HWLOC_SLEVEL_FROM_DEPTH(src->depth);
     level = newtopology->slevels[i].objs;
+    level_width = newtopology->slevels[i].nbobjs;
     /* deal with first/last pointers of special levels, even if not really needed */
     if (!newobj->logical_index)
       newtopology->slevels[i].first = newobj;
@@ -718,13 +719,22 @@
       newtopology->slevels[i].last = newobj;
   } else {
     level = newtopology->levels[src->depth];
+    level_width = newtopology->level_nbobjects[src->depth];
   }
-  /* place us for real and link to previous cousin */
+  /* place us for real */
+  assert(newobj->logical_index < level_width);
   level[newobj->logical_index] = newobj;
-  if (newobj->logical_index) {
+  /* link to already-inserted cousins
+   * (hwloc_pci_belowroot_apply_locality() can cause out-of-order logical indexes)
+   */
+  if (newobj->logical_index > 0 && level[newobj->logical_index-1]) {
     newobj->prev_cousin = level[newobj->logical_index-1];
     level[newobj->logical_index-1]->next_cousin = newobj;
   }
+  if (newobj->logical_index < level_width-1 && level[newobj->logical_index+1]) {
+    newobj->next_cousin = level[newobj->logical_index+1];
+    level[newobj->logical_index+1]->prev_cousin = newobj;
+  }
 
   /* prepare for children */
   if (src->arity) {

[rhc54]

@bgoglin what branch should I pull this from? The patch doesn't apply cleanly to what I have, so I assume you have an updated branch I should use?

[bgoglin]

Strange, it applies on top of OMPI master here:

$ git describe 
v2.x-dev-4604-g628b4f7add
$ patch -p1 --dry-run < ~/fix-duplicate.patch
checking file hwloc/topology.c
Hunk #1 succeeded at 651 (offset -5 lines).
Hunk #2 succeeded at 701 (offset -11 lines).
Hunk #3 succeeded at 709 (offset -11 lines).

I'll update my shmem branch with this and other hwloc git master changes once the bug fix is confirmed.

[rhc54]

It's the hunk offsets that triggered my comment - if things were up-to-date, that shouldn't happen