3.0.0rc2: FreeBSD: divide-by-zero in hwloc

[hppritcha]

@PHHargrove reported a problem bubbling up from hwloc when testing 3.0.0rc2 on
FreeBSD/amd64:

On FreeBSD-11/amd64:

$ mpirun -mca btl vader,self -np 2 examples/ring_c
[freebsd-amd64:25312] *** Process received signal ***
[freebsd-amd64:25312] Signal: Floating point exception (8)
[freebsd-amd64:25312] Signal code: Integer divide-by-zero (2)
[freebsd-amd64:25312] Failing at address: 0x800c13822
[freebsd-amd64:25312] [ 0] 0x8016e5934 <pthread_sigmask+0x544> at /lib/libthr.so.3
[freebsd-amd64:25312] [ 1] 0x8016e4ecf <pthread_getspecific+0xe5f> at /lib/libthr.so.3
[freebsd-amd64:25312] *** End of error message ***

reported on devel mail list:
https://www.mail-archive.com/devel@lists.open-mpi.org//msg20326.html

[PHHargrove]

If I configure using --with-hwloc=/usr/local I get hwloc-1.11.7 (newer than the embedded 1.11.3) and the problem does not occur. The FreeBSD ports tree does not appear to be applying any patches of its own. So, there is potentially an upstream fix that can be cherry-picked.

Alternatively, README could suggest installing/using an external hwloc.

In case it matters, this is a KVM-based virtual machine.
Output of lstopo (from FreeBSD's 1.11.7):

$ lstopo -v
Machine (P#0 local=3112740KB total=3112740KB Backend=FreeBSD OSName=FreeBSD OSRelease=11.1-RELEASE OSVersion="FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC" HostName=freebsd-amd64 Architecture=amd64 Backend=x86 hwlocVersion=1.11.7 ProcessName=lstopo)
  Package L#0 (P#0 CPUVendor=AuthenticAMD CPUFamilyNumber=6 CPUModelNumber=6 CPUModel="QEMU Virtual CPU version 2.5+" CPUStepping=3)
    L2Cache L#0 (P#0 size=512KB linesize=64 ways=16 Inclusive=0)
      L1dCache L#0 (P#0 size=64KB linesize=64 ways=2 Inclusive=0)
        L1iCache L#0 (P#0 size=64KB linesize=64 ways=2 Inclusive=0)
          Core L#0 (P#0)
            PU L#0 (P#0)
  Package L#1 (P#1 CPUVendor=AuthenticAMD CPUFamilyNumber=6 CPUModelNumber=6 CPUModel="QEMU Virtual CPU version 2.5+" CPUStepping=3)
    L2Cache L#1 (P#1 size=512KB linesize=64 ways=16 Inclusive=0)
      L1dCache L#1 (P#1 size=64KB linesize=64 ways=2 Inclusive=0)
        L1iCache L#1 (P#1 size=64KB linesize=64 ways=2 Inclusive=0)
          Core L#1 (P#0)
            PU L#1 (P#1)
depth 0:        1 Machine (type #1)
 depth 1:       2 Package (type #3)
  depth 2:      2 L2Cache (type #4)
   depth 3:     2 L1dCache (type #4)
    depth 4:    2 L1iCache (type #4)
     depth 5:   2 Core (type #5)
      depth 6:  2 PU (type #6)

[bwbarrett]

Interesting; I can't replicate on EC2 with 11.0; wonder if something changed between 11.0 and 11.1 or if it's something in the QEMU processor emulation.

$ mpirun -mca btl vader,self -np 2 ./ring_c
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting
$ lstopo -v
Machine (P#0 local=31424180KB total=31424180KB Backend=FreeBSD OSName=FreeBSD OSRelease=11.0-RELEASE-p9 OSVersion="FreeBSD 11.0-RELEASE-p9 #0: Tue Apr 11 08:48:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC" HostName=ip-172-31-57-222 Architecture=amd64 Backend=x86 hwlocVersion=1.11.7 ProcessName=lstopo)
  Package L#0 (P#0 CPUVendor=GenuineIntel CPUFamilyNumber=6 CPUModelNumber=63 CPUModel="Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz" CPUStepping=2)
    L3Cache L#0 (P#0 size=25600KB linesize=64 ways=20 Inclusive=1)
      L2Cache L#0 (P#0 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#0 (P#0 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#0 (P#0 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#0 (P#0)
              PU L#0 (P#0)
              PU L#1 (P#1)
      L2Cache L#1 (P#1 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#1 (P#1 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#1 (P#1 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#1 (P#1)
              PU L#2 (P#2)
              PU L#3 (P#3)
      L2Cache L#2 (P#2 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#2 (P#2 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#2 (P#2 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#2 (P#2)
              PU L#4 (P#4)
              PU L#5 (P#5)
      L2Cache L#3 (P#3 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#3 (P#3 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#3 (P#3 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#3 (P#3)
              PU L#6 (P#6)
              PU L#7 (P#7)
      L2Cache L#4 (P#4 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#4 (P#4 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#4 (P#4 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#4 (P#4)
              PU L#8 (P#8)
              PU L#9 (P#9)
      L2Cache L#5 (P#5 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#5 (P#5 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#5 (P#5 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#5 (P#5)
              PU L#10 (P#10)
              PU L#11 (P#11)
      L2Cache L#6 (P#6 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#6 (P#6 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#6 (P#6 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#6 (P#6)
              PU L#12 (P#12)
              PU L#13 (P#13)
      L2Cache L#7 (P#7 size=256KB linesize=64 ways=8 Inclusive=0)
        L1dCache L#7 (P#7 size=32KB linesize=64 ways=8 Inclusive=0)
          L1iCache L#7 (P#7 size=32KB linesize=64 ways=8 Inclusive=0)
            Core L#7 (P#7)
              PU L#14 (P#14)
              PU L#15 (P#15)
depth 0:	1 Machine (type #1)
 depth 1:	1 Package (type #3)
  depth 2:	1 L3Cache (type #4)
   depth 3:	8 L2Cache (type #4)
    depth 4:	8 L1dCache (type #4)
     depth 5:	8 L1iCache (type #4)
      depth 6:	8 Core (type #5)
       depth 7:	16 PU (type #6)

[rhc54]

Is there any reason not to update to 1.11.7? I ask because there are some improvements and bug fixes in those latter releases that are good to have. I'm willing to roll the PR if you want it.

[bwbarrett]

@rhc54, I don't think so; just the usual wanting to get something out the door. But if that's the easiest way to get this bug removed, then let's do that.

[PHHargrove]

I've tried #4022, but the problem persists. Perhaps I was wrong when I concluded that no patches were applied to the hwloc build in the FreeBSD package repo.

The fastest way to get 3.0 out the door seems to me to be to simply document that --with-hwloc=/usr/local is necessary. Unfortunately, beween Brian's FreeBSD-11.0+EC2 results and my FreeBSD-11.1+KVM results, we are left uncertain if the O/S or QEMU is the key factor.

If one does want to continue chasing this bug, here is something new... it looks like the hwloc cache is all zeros at the time of the crash:

#0  0x0000000800c1395e in look_proc (backend=0x8026264a0, infos=0x802690500, highest_cpuid=13,
    highest_ext_cpuid=2147483658, features=0x7fffffffdcd0, cpuid_type=57921536)
    at ../../../../../../../../opal/mca/hwloc/hwloc1117/hwloc/src/topology-x86.c:482
482         cache->cacheid = infos->apicid / cache->nbthreads_sharing;
(gdb) print *cache
$1 = {type = 0, level = 0, nbthreads_sharing = 0, cacheid = 0, linesize = 0, linepart = 0, inclusive = 0,  ways = 0, sets = 0, size = 0}

[rhc54]

@bgoglin Any thoughts?

[bgoglin]

I am not aware of any patch being applied to FreeBSD hwloc.

@PHHargrove when cache->type == 0, we ignore the cache, you shouldn't ever use that cache in the code. I have a FreeBSD 11.1 running on KVM, no problem. But it may depend on the underlying hardware since we are in the cpuid/x86 code here. Can you run hwloc-gather-cpuid from hwloc git master and send the resulting cpuid directory? I can debug remotely from there.

[PHHargrove]

@bgoglin
hwloc-gather-cpuid from hwloc git master SEGVs:

Program terminated with signal 11, Segmentation fault.
Reading symbols from /home/phargrov/hwloc/BLD/hwloc/.libs/libhwloc.so.0...done.
Loaded symbols for /home/phargrov/hwloc/BLD/hwloc/.libs/libhwloc.so.0
[...]
#0  hwloc_look_x86 (backend=0x802040050, fulldiscovery=<value optimized out>) at ../../hwloc/topology-x86.c:63
63        if (cpuiddump->nr)
(gdb) where
#0  hwloc_look_x86 (backend=0x802040050, fulldiscovery=<value optimized out>) at ../../hwloc/topology-x86.c:63
#1  0x000000080084cf87 in hwloc_x86_discover (backend=0x802040050) at ../../hwloc/topology-x86.c:1257
#2  0x000000080082fafa in hwloc_topology_load (topology=0x80201c000) at ../../hwloc/topology.c:2609
#3  0x000000000040115f in main (argc=<value optimized out>, argv=0x7fffffffe9f0)
    at ../../../utils/hwloc/hwloc-gather-cpuid.c:378

[bgoglin]

Ergl, it likely fails for similar reasons. Can you set HWLOC_COMPONENTS=-x86 in the environment before running?