Profile Native Extensions

[benfred]

Right now we are figuring out the call stack of the Python program by inspecting the PyInterpreterState/PyThreadState/PyFrameObject variables found in the target python program. This means we don't get information about time spent in non-python threads and time spent in native extensions to python (like extension code written in Cython or C++)

It could be useful to profile these native extensions in addition to the Python call stacks. It might be possible to use something like libunwind to augment the Python call stack to get this information.

edit: There is a prerelease here pip install py-spy==0.2.0.dev4 that can profile native extensions of 64-bit Linux/Windows.

[anntzer]

You may(?) want to compare with https://github.com/vmprof/vmprof-python, which also uses libunwind.

[okapies]

I also eager to have the native backtrace support in py-spy! It seems this is a secret sauce of vmprof. It depends on libbacktrace which uses libunwind internally.

https://github.com/vmprof/vmprof-python/blob/bb40a6873669ed414817f87b4a0cde5d5b90af82/src/symboltable.c#L213-L267

[nikhilm]

This is something I've mostly thought about in the context of a native sampling profiler 1 systematically detecting Python code while unwinding, and interleaving python info at that point, but not actually tried.

The big problem is reconciling the state of the world from the interpreter, to that of the native stack, as frames are created and destroyed. The naive way is to rescan the interpreter "tree" every time one encounters a PyEval function on the stack, and then literally counting the frame list and the unwind stack to find which frame corresponds to this native function. The better way is to know the PyFrameObject directly. It is basically impossible under x86_64 to retrieve the PyFrameObject from a PyEval_EvalFrameEx call while unwinding because it is not required to be in a non-clobbered register. On x86 (32-bit), you can scan the stack and match again the tree inferred from the interpreter state, but this is slow/error prone.

PEP 523 (Py 3.6 onwards) introduced the ability to replace PyEval_EvalFrameDefault with a custom function. This can be done by simply loading a custom python extension in the interpreter. Once you have this, you are free to stash the frame pointer in a non-clobbered register (callee/caller-saved? I'm always confused), before continuing frame execution. At profile collection time, when you are unwinding the stacks, I conjecture that there is enough unwind info to retrieve this register content. If you do try this out, let me know. I'm very curious to know if this works :). vmprof doesn't use it because it is a relatively new thing, and it does require that you ship/use a python extension with your intepreter that injects itself to replace this function.

[nikhilm]

vmprof stops at the first PyEvalFrame call and switches entirely to Python
https://github.com/vmprof/vmprof-python/blob/e9c52d9131645879a3bcdd2abcbea70b384fe544/src/vmp_stack.c#L295
Their documentation seems to state there was an earlier implementation using PEP 523 where available and a custom trampoline at other times. https://vmprof.readthedocs.io/en/latest/native.html#technical-design
I have not searched the historical code.

[benfred]

Thanks for the pointers @nikhilm and @anntzer! I hadn't seen PEP 523 before.

Just as a quick update on this, I’ve basically got this working for 64bit Linux and OSX right now. It needs a bunch more cleanup, but I should have a initial version posted here in the next little bit. I think windows should be also relatively straightforward to do with the StackWalk64 API but I haven't started on that yet.

I don't want to modify the target program and also want to support profiling older versions of Python - so while using PEP 523 it is a cool idea I don't think it would work here. Like @nikhilm, I also found it tricky to reliably get the PyFrameObject pointer from the call stack with x86_64 calling conventions (it’s passed in to PyEval_EvalFrame* on the RDI register which isn’t unwound =(.

So instead I'm getting both the interpreter stack and the native stack - and then merging the two stack's together by replacing the native PyEval_EvalFrame.* calls with the corresponding python frame.

Like given this python stack:

Thread 0x7FFF98D36380 (active)
fit (implicit/als.py:163)
calculate_similar_artists (lastfm.py:87)
<module> (lastfm.py:186)

And this corresponding native stack with native and Cython code:

Thread 2563
0x00007fff34dfacf0 _cblas_saxpy (/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib)
0x00007fff34def7b0 _SAXPY (/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib)
0x000000010b0d8a9a __ZL44__pyx_pf_8implicit_4_als_30_least_squares_cgP7_object18__Pyx_memviewsliceS1_S1_S1_S1_fii._omp_fn.9 (implicit/_als.cpp:15554)
0x000000010b47be5f _GOMP_parallel (/usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib)
0x000000010b10c0b6 __ZL58__pyx_fuse_1_0__pyx_pw_8implicit_4_als_31_least_squares_cgP7_objectS0_S0_ (implicit/_als.cpp:15812)
0x000000010b0e5b23 __ZL24__pyx_FusedFunction_callP7_objectS0_S0_ (implicit/_als.cpp:43560)
0x000000010b0d3acc __ZL19__Pyx_PyObject_CallP7_objectS0_S0_ (implicit/_als.cpp:40769)
0x000000010b0ea96a __ZL42__pyx_pw_8implicit_4_als_5least_squares_cgP7_objectS0_S0_ (implicit/_als.cpp:11621)
0x0000000108f1d12b __PyCFunction_FastCallDict (/Users/ben/anaconda3/bin/python)
0x0000000108e9b8c3 __PyObject_FastCallDict (/Users/ben/anaconda3/bin/python)
0x00000001090a3026 _partial_call (/Users/ben/anaconda3/bin/python)
0x0000000108e9b821 __PyObject_FastCallDict (/Users/ben/anaconda3/bin/python)
0x0000000108ea3a67 __PyObject_FastCallKeywords (/Users/ben/anaconda3/bin/python)
0x0000000108ff4048 _call_function (/Users/ben/anaconda3/bin/python)
0x0000000108ff1b6f __PyEval_EvalFrameDefault (/Users/ben/anaconda3/bin/python)
0x0000000108fe5209 __PyEval_EvalCodeWithName (/Users/ben/anaconda3/bin/python)
0x0000000108ff43ba _fast_function (/Users/ben/anaconda3/bin/python)
0x0000000108ff3fac _call_function (/Users/ben/anaconda3/bin/python)
0x0000000108ff1abf __PyEval_EvalFrameDefault (/Users/ben/anaconda3/bin/python)
0x0000000108fe5209 __PyEval_EvalCodeWithName (/Users/ben/anaconda3/bin/python)
0x0000000108ff43ba _fast_function (/Users/ben/anaconda3/bin/python)
0x0000000108ff3fac _call_function (/Users/ben/anaconda3/bin/python)
0x0000000108ff1b6f __PyEval_EvalFrameDefault (/Users/ben/anaconda3/bin/python)
0x0000000108fe5209 __PyEval_EvalCodeWithName (/Users/ben/anaconda3/bin/python)
0x000000010903dd4c _PyRun_FileExFlags (/Users/ben/anaconda3/bin/python)
0x000000010903d224 _PyRun_SimpleFileExFlags (/Users/ben/anaconda3/bin/python)
0x0000000109063d66 _Py_Main (/Users/ben/anaconda3/bin/python)
0x0000000108e93929 _main (/Users/ben/anaconda3/bin/python)
0x00007fff60233015 _start (/usr/lib/system/libdyld.dylib)

This code merges them together to produce output that looks like

Thread 0x7FFF98D36380/2563 (active)
_cblas_saxpy (libBLAS.dylib)
_SAXPY (libBLAS.dylib)
_least_squares_cg (implicit/_als.pyx:200)
_GOMP_parallel (libgomp.1.dylib)
_least_squares_cg (implicit/_als.pyx:156)
least_squares_cg (implicit/_als.pyx:135)
fit (implicit/als.py:163)
calculate_similar_artists (lastfm.py:87)
<module> (lastfm.py:186)

There are a bunch of details I’m skipping over (matching the OS thread id and the python thread id, demangling Cython etc), and a bunch of other things that still need done - but I don’t see any blocking issues at this stage. I’ll probably write a blog post about this at some point with more details.

Also, I’m using libunwind only as a fallback, and instead using gimli to unwind (which seems to be about 10x faster).

[nikhilm]

Looks awesome! Exactly what I was thinking.
If PEP-523 isn't viable, and this approach is slow, then you may want to do what vmprof used to do, which is to replace the first few instructions of PyEval_EvalFrameEx with a custom piece of code, move the existing instructions into a different memory page and then stash rdi into rbx or similar. I believe pyflame has some examples of arbitrary code execution in the remote process using ptrace, which is Linux only.

I'm curious about libunwind vs gimli. Are you using the libunwind-sys APIs directly, or the higher level libunwind? I wasn't aware that gimli had a direct unwinding API, i thought it was just a parser. One thing to keep in mind is that macOS often uses compact unwind encoding for unwind information, which afaik is not DWARF.

As for using StackWalk64, it may not be the fastest for a profiler, as it was always made for debugging, and tends to want to lookup symbols. I believe this is why Firefox doesn't use it as the first choice in their profiler. On x86_64, there is RtlUnwind on windows, which is supposed to be much faster. On x86, you can do direct rbp/rsp based unwinding, with some considerations for leaf functions.

[benfred]

For OSX, I wrote my own compact unwind code in Rust - and am using gimli for the dwarf unwinding part.

For Linux, gimli doesn't provide a unwind API directly but it's pretty easy to build one on top of it (it's only a couple of hundred lines of code). I'm using libunwind-ptrace for a fallback in case the gimli unwinder fails - I've enabled caching w/ the _Ux86_64_set_caching_policy call, but it is still pretty slow compared to gimli. I'll post the code here soon - I'd be interested in your feedback on it.

I hadn't seen RtlUnwind for windows yet- thanks for the heads up!

[nikhilm]

I'm glad you have a lot of free time! :D

[nikhilm]

Ah! Ok, I've mostly been thinking of in-process profiling, where everything is much faster :)
Yes, in that case you'd have to write your own for OSX or use PLCrashReporter's code 1 since libunwind on OSX doesn't support remote unwinding. I am also not sure if RtlUnwindEx and co will work directly then.

[benfred]

I've got a first draft of this here: #85 . For windows, I went with the StackWalk64 api since I don't think that RtlUnwindEx will work with a remote process.

It still needs some work, but since it doesn't really impact existing functionality (only is called when you pass --native from commandline) I'm leaning towards pushing out a dev release so that people can start playing around with it.

edit: you can install the dev release on linux/osx by going pip install py-spy==0.2.0.dev0

[belm0]

observations from using --native on dev build:

1. py-spy gave warning about libunwind, but I already have it installed

Failed to load libunwind-ptrace, you may have an elevated error rate
You can install libunwind-ptrace on ubuntu by going 'sudo apt install libunwind-dev'

$ apt-cache policy libunwind-dev
libunwind-dev:
  Installed: 1.1-4.1
  Candidate: 1.1-4.1

2. error on py-spy start

ERROR 2019-02-26T04:34:10Z: remoteprocess::linux::symbolication: failed to parse file for symbolication /.../lib/python3.7/site-packages/numpy/core/multiarray.cpython-37m-x86_64-linux-gnu.so: "Could not parse ELF header"

3. high sampling error rate (~30%)
   errors: NoUnwindInfoForAddress, Failed to merge native and python frames (Have 0 native and 12 python)