Optionally include lower level stack information in profile plots

[mrocklin]

The current statistical profiler can provide a ton of useful information about where user code spends time. Unfortunately it only looks down through Python code. Anything lower level is effectively opaque. For people wanting to profile low level compiled code this can be a limitation.

Last year @sklam integrated libunwind with stacktrace into Dask's statistical profiler here: https://github.com/dask/distributed/compare/master...sklam:stacktrace_integration?expand=1

Some work to do:

1. Fix merge conflicts (probably simple)
2. Add a test or two?
3. Make it optional with configuration
4. See how much performance cost arises from its use
5. Try it out, report feedback, fix bugs

[mrocklin]

@beberg this may interest you

[pentschev]

I think items 1, 2 and 3 were properly tackled by #2575. As for bugs (item 5), the only known bug is described in #2608.

Regarding performance, from my tests I saw low-level profiler almost doubling execution time in some cases, which doesn't really surprise me a lot, since a much larger amount of functions can be expected to derive from C calls. However, I don't know if this information belongs somewhere, perhaps a warning in documentation?

Other than that, I think the scope of this issue has been fully tackled, so I think we can close this too.

[mrocklin]

This is in. Thanks @pentschev