Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid inlining exact_unwind #52

Merged
merged 1 commit into from May 17, 2022
Merged

Conversation

matthiasdiener
Copy link
Contributor

@matthiasdiener matthiasdiener commented Apr 22, 2022

Issue number of the reported bug or feature request: #51

Describe your changes
Remove inline attribute from exact_unwind()

Testing performed
Compiled and ran a simple test on a ppc64le machine.

@matthiasdiener matthiasdiener changed the title remove inline attribute from exact_unwind for ppc64le remove inline attribute from exact_unwind for compiling on ppc64le Apr 22, 2022
@pablogsal
Copy link
Member

pablogsal commented Apr 22, 2022

One possibility is trying to ensure this is never inlined and adjusting the skip parameter:

size_t d_skip = 0;

Or another way is transforming the function into a macro as @godlygeek suggested offline.

@godlygeek
Copy link
Contributor

godlygeek commented Apr 22, 2022

This patch seems like it ought to do the trick - I tested for a few moments locally, and it seems to pass the test suite both for a -O3 and a -O0 build of the interpreter, at least on x86_64. Can you try it out on ppc64le?

diff --git a/src/memray/_memray/tracking_api.h b/src/memray/_memray/tracking_api.h
index 8118b43..06ab288 100644
--- a/src/memray/_memray/tracking_api.h
+++ b/src/memray/_memray/tracking_api.h
@@ -89,6 +89,7 @@ class NativeTrace
         if (size == MAX_SIZE) {
             d_data.resize(0);
             size = exact_unwind();
+            skip += 1;  // skip the non-inlined exact_unwind frame
             MAX_SIZE = MAX_SIZE * 2 > size ? MAX_SIZE * 2 : size;
             d_data.resize(MAX_SIZE);
         }
@@ -122,7 +123,7 @@ class NativeTrace
         return unw_backtrace((void**)data, MAX_SIZE);
     }

-    __attribute__((always_inline)) size_t inline exact_unwind()
+    __attribute__((noinline)) size_t inline exact_unwind()
     {
         unw_context_t context;
         if (unw_getcontext(&context) < 0) {

@matthiasdiener
Copy link
Contributor Author

This patch seems like it ought to do the trick - I tested for a few moments locally, and it seems to pass the test suite both for a -O3 and a -O0 build of the interpreter, at least on x86_64. Can you try it out on ppc64le?

diff --git a/src/memray/_memray/tracking_api.h b/src/memray/_memray/tracking_api.h
index 8118b43..06ab288 100644
--- a/src/memray/_memray/tracking_api.h
+++ b/src/memray/_memray/tracking_api.h
@@ -89,6 +89,7 @@ class NativeTrace
         if (size == MAX_SIZE) {
             d_data.resize(0);
             size = exact_unwind();
+            skip += 1;  // skip the non-inlined exact_unwind frame
             MAX_SIZE = MAX_SIZE * 2 > size ? MAX_SIZE * 2 : size;
             d_data.resize(MAX_SIZE);
         }
@@ -122,7 +123,7 @@ class NativeTrace
         return unw_backtrace((void**)data, MAX_SIZE);
     }

-    __attribute__((always_inline)) size_t inline exact_unwind()
+    __attribute__((noinline)) size_t inline exact_unwind()
     {
         unw_context_t context;
         if (unw_getcontext(&context) < 0) {

Thanks, that looks good!

Many pytest tests are still failing, but the basic functionality seems to work fine.

@pablogsal
Copy link
Member

Many pytest tests are still failing, but the basic functionality seems to work fine.

Even with the patch? Mind sharing the pytest output?

@matthiasdiener
Copy link
Contributor Author

Many pytest tests are still failing, but the basic functionality seems to work fine.

Even with the patch? Mind sharing the pytest output?

Yes, even with the patch. pytest segfaults pretty early during the run:

$ pytest
============================================================ test session starts ============================================================
platform linux -- Python 3.9.10, pytest-7.0.0, pluggy-1.0.0
rootdir: /usr/WS1/diener3/Work/src/memray, configfile: pyproject.toml
plugins: pudb-0.7.0, cov-3.0.0
collected 333 items

tests/test_utils.py .......                                                                                                           [  2%]
tests/integration/test_api.py ......                                                                                                  [  3%]
tests/integration/test_extensions.py F....                                                                                            [  5%]
tests/integration/test_main.py ............FF.FFFFFF.....FF...........F.                                                              [ 17%]
tests/integration/test_native_tracking.py FFatal Python error: Segmentation fault

Current thread 0x0000200000044b10 (most recent call first):
  File "/usr/WS1/diener3/Work/src/memray/tests/integration/test_native_tracking.py", line 90 in test_simple_call_chain_with_native_tracking
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/python.py", line 1718 in runtest
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 261 in <lambda>
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 340 in from_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 322 in _main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/bin/pytest", line 11 in <module>
Segmentation fault

@pablogsal
Copy link
Member

Many pytest tests are still failing, but the basic functionality seems to work fine.

Even with the patch? Mind sharing the pytest output?

Yes, even with the patch. pytest segfaults pretty early during the run:

$ pytest
============================================================ test session starts ============================================================
platform linux -- Python 3.9.10, pytest-7.0.0, pluggy-1.0.0
rootdir: /usr/WS1/diener3/Work/src/memray, configfile: pyproject.toml
plugins: pudb-0.7.0, cov-3.0.0
collected 333 items

tests/test_utils.py .......                                                                                                           [  2%]
tests/integration/test_api.py ......                                                                                                  [  3%]
tests/integration/test_extensions.py F....                                                                                            [  5%]
tests/integration/test_main.py ............FF.FFFFFF.....FF...........F.                                                              [ 17%]
tests/integration/test_native_tracking.py FFatal Python error: Segmentation fault

Current thread 0x0000200000044b10 (most recent call first):
  File "/usr/WS1/diener3/Work/src/memray/tests/integration/test_native_tracking.py", line 90 in test_simple_call_chain_with_native_tracking
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/python.py", line 1718 in runtest
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 261 in <lambda>
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 340 in from_call
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 322 in _main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/lib/python3.9/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/usr/WS1/diener3/Work/emirge/miniforge3/envs/ceesd/bin/pytest", line 11 in <module>
Segmentation fault

Wow, any chance you can give us a stack trace with gdb?

@matthiasdiener
Copy link
Contributor Author

Wow, any chance you can give us a stack trace with gdb?

Sure, here is the trace:

tests/integration/test_native_tracking.py [Detaching after fork from child process 98675]
[New Thread 0x20000288f1b0 (LWP 98972)]

Program received signal SIGSEGV, Segmentation fault.
0x00002000020debdc in memray::tracking_api::Tracker::trackAllocationImpl(void*, unsigned long, memray::hooks::Allocator) ()
   from /usr/WS1/diener3/Work/src/memray/src/memray/_memray.cpython-39-powerpc64le-linux-gnu.so
(gdb) bt
#0  0x00002000020debdc in memray::tracking_api::Tracker::trackAllocationImpl(void*, unsigned long, memray::hooks::Allocator) ()
   from /usr/WS1/diener3/Work/src/memray/src/memray/_memray.cpython-39-powerpc64le-linux-gnu.so
#1  0x00002000020b6cb4 in memray::intercept::mmap(void*, unsigned long, int, int, int, long) ()
   from /usr/WS1/diener3/Work/src/memray/src/memray/_memray.cpython-39-powerpc64le-linux-gnu.so
#2  0x00002000000c9c70 in allocate_stack (stack=<synthetic pointer>, pdp=<synthetic pointer>, attr=0x2000000f4428 <__default_pthread_attr>)
    at allocatestack.c:486
#3  __pthread_create_2_1 (newthread=0x7fffffff43c0, attr=0x0, start_routine=0x200002450dc0 <worker(void*)>, arg=0x0) at pthread_create.c:447
#4  0x0000200002450f64 in start_threads() ()
   from /tmp/pytest-of-diener3/pytest-13/test_multithreaded_extension_w0/multithreaded_extension/testext.cpython-39-powerpc64le-linux-gnu.so
#5  0x0000200002451038 in run(_object*, _object*) ()
   from /tmp/pytest-of-diener3/pytest-13/test_multithreaded_extension_w0/multithreaded_extension/testext.cpython-39-powerpc64le-linux-gnu.so
#6  0x00000001002c49c8 in cfunction_vectorcall_NOARGS ()
[...]

@pablogsal
Copy link
Member

pablogsal commented Apr 22, 2022

Hummm, could you compile with -O0 -g3 and try again to get the backtrace to get the biggest amount of DWARF info so hopefully we can get the line number where this happened?

Alternatively (or additionally), run "disas" at the segfault to know what is going on with the instructions there.

I suspect this may be the libunwind call but it may be the TLS as well.

@godlygeek
Copy link
Contributor

Taking a slightly closer look at that stack: it's glibc (good, I was afraid it might be musl). It's in the middle of a call to pthread_create, making a new thread. In particular, it's in the middle of a call to allocate_stack, which is creating the stack that the new thread will use once it runs - but, the new thread hasn't actually been clone()d yet, we're still in the parent thread (which is obvious from the stack trace, in retrospect).

Based on this, I agree - it's more likely that libunwind has failed us than that TLS is the problem (I would have suspected TLS if we were in the new thread instead)

@godlygeek godlygeek force-pushed the ppc64le branch 3 times, most recently from 16a8d36 to 1b347ab Compare May 16, 2022 23:26
On some architectures this can't be inlined. We don't actually care
whether or not it is inlined, only that whether it is or isn't inlined
is consistent regardless of build flags.

Signed-off-by: Matthias Diener <mdiener@illinois.edu>
Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
@godlygeek godlygeek changed the title remove inline attribute from exact_unwind for compiling on ppc64le Avoid inlining exact_unwind May 16, 2022
@godlygeek godlygeek enabled auto-merge (rebase) May 16, 2022 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants