can't unwind out of pthread_mutex_lock on powerpc

[mwkrentel]

I updated my old malloc stress test to use threads. I was expecting
to be able to deadlock libunwind pretty easily, but fortunately that
doesn't seem to be the case. (The Machine takes a hit. :-)
However, it did reveal two other problems.

1. On powerpc (llnl.cs, power9), we can't unwind out of
   pthread_mutex_lock(). As a result, I'm getting about 25% partial
   unwinds.

UNWIND ANOMALIES: total: 932415 errant: 208475, total-frames: 4306428, total-libunwind-fails: 0
SAMPLE ANOMALIES: blocks: 0 (async: 0, dlopen: 0), errors: 208475 (segv: 0, soft: 208475)
SUMMARY: samples: 932415 (recorded: 932415, blocked: 0, errant: 208475, trolled: 0, yielded: 0),

Btw, I see that for REALTIME 'usec' for micro-seconds has been changed
to 's'. I think 's' is too ambiguous and 'sec' for seconds would be
more clear to the non-expert.

---

2. On x86_64, I get a smattering of segvs (ones that we catch and
   continue on), all in vdso. Maybe about 1 per 200,000 or 300,000
   interrupts.

[25111, 4]: error: segv: context-pc=0x7ff40654e31c
[25111, 4]: -- begin new backtrace (innermost first) [SEGV] ----------
[25111, 4]: NULL (0): ip = 0x7fffe5cf8d3e (0xd3e), load module = 
/home/krentel/memstress/measure-64/vdso/f9ba0ffc66627576389c1a9bb2438f2d.[vdso] | lip 
[25111, 4]: -- end backtrace ------------------------------------

[25111, 7]: error: segv: context-pc=0x7ff40654e31c
[25111, 7]: -- begin new backtrace (innermost first) [SEGV] ----------
[25111, 7]: NULL (0): ip = 0x7fffe5cf8d3e (0xd3e), load module = 
/home/krentel/memstress/measure-64/vdso/f9ba0ffc66627576389c1a9bb2438f2d.[vdso] | lip 
[25111, 7]: -- end backtrace ------------------------------------

[25111, 1]: error: segv: context-pc=0x7ff40654e31c
[25111, 1]: -- begin new backtrace (innermost first) [SEGV] ----------
[25111, 1]: NULL (0): ip = 0x7fffe5cf8d3e (0xd3e), load module = 
/home/krentel/memstress/measure-64/vdso/f9ba0ffc66627576389c1a9bb2438f2d.[vdso] | lip 
[25111, 1]: -- end backtrace ------------------------------------

UNWIND ANOMALIES: total: 901748 errant: 6, total-frames: 5161706, total-libunwind-fails: 0
SAMPLE ANOMALIES: blocks: 1 (async: 1, dlopen: 0), errors: 6 (segv: 6, soft: 0)
SUMMARY: samples: 901748 (recorded: 901747, blocked: 1, errant: 6, trolled: 0, yielded: 0),
         frames: 5161706 (trolled: 0)
         intervals: 375 (suspicious: 0)

---

I'll attach a version of my unit test. It's easy to compile (with
-lpthread) and run. I was running with REALTIME at a fairly fast
clip. In this example, it runs for 30 seconds with 8 threads and 4000
interrupts per second per thread.

hpcrun -e REALTIME@250 ./memstress 30 8

Both of these are easy to reproduce, so no real need to save
and point to a specific example.

[mwkrentel]

Source file.
memstress.c.gz

[mwkrentel]

Things have gotten worse since the gpu merge. Before the merge, from
0d91fb34edb2, I get about 0.01% partial call paths and a different
log message. With the same conditions:

[128911, 2]: unw_init: cursor could NOT build an interval for initial pc = 0x7fff8057066c
[128911, 3]: unw_init: cursor could NOT build an interval for initial pc = 0x7fff8057066c
[128911, 6]: unw_init: cursor could NOT build an interval for initial pc = 0x7fff805706d8
[128911, 7]: unw_init: cursor could NOT build an interval for initial pc = 0x7fff80570684
MEMORY: segment size: 4.0 meg, num segments: 9, total allocation: 36.0 meg, reclaims: 0
MEMORY: total freeable: 0.0 meg, total non-freeable: 2.5 meg, malloc failures: 0
SAMPLE ANOMALIES: blocks: 0 (async: 0, dlopen: 0), errors: 94 (segv: 0, soft: 94)
SUMMARY: samples: 913900 (recorded: 913900, blocked: 0, errant: 94, trolled: 0, yielded: 0),
         frames: 5039656 (trolled: 0)
         intervals: 0 (suspicious: 0)

[jmellorcrummey]

fixed by PR #230