Seeing Segmentation fault usign pyroscope-io python package for profiling a python app

[venu-shapley]

We are using pyroscope-io package to profile a python application which is a fastAPI server.

During the initialization, I configure the pyroscope but I am seeing Segmentation faults and pointed to the following difference when the service starts up vs running into a segmentation fault.

Can you please help me understand what is going on?

[korniltsev]

What is the pip package version? Does it crash without pyroscope?

Just to clarify, does it crash every time at the startup? or some times it does not crash?

Can you run the python app with env RUST_BACKTRACE=full ?

Another thing that could help in theory is full debug which you can enable like this

l = logging.getLogger()
l.setLevel(logging.DEBUG)

pyroscope.configure(
# ...
	enable_logging = True,
)

Can you attach lldb to a running python process like this lldb -p $(echo $YOUR_PYTHON_PID) --one-line continue and then when you get the Segmentation fault in the debugger you could run two commands bt and reg read - that could give some starting point for investigation. And then share the backtrace and regs dump.
Or if the crash happens too quickly before you can attach with the lldb debugger, you can start it from the debugger like this lldb /Users/korniltsev/.pyenv/versions/3.11.9/bin/python3 --one-line 'run your_script.py' and again once you get the sgementation fault do bt and reg read and share here.
You may need adjust the commands little for your app.

[venu-shapley]

The package version we are using is pyroscope-io = "0.8.7".
It doesn't crash when I don't initialize the pyroscope profiler.
It doesn't crash everytime, it does it sometimes, but it is not rare to crash.
The crash is happening very early, even before the fast api server is ready to serve requests. I'll try to get the lldb debug info shortly.

[ThisIsANiceName]

I got the same issue with the 0.8.7 version.

As mentioned before it doesn't happen every time.

Here is the backtrace from gdb in case this helps. (I did not do more investigration into the issue, and downgraded to 0.8.5)

[vishal-wadhwa]

Facing similar issue
RUST backtrace
MacOS 14.5 (Sonoma) M1

 2024-06-03T08:09:47.973Z WARN  py_spy::sampler > Failed to open process 66517: Undefined error: 0 (os error 0)
 2024-06-03T08:09:48.012Z WARN  py_spy::sampler > Failed to create spy for 66517: Undefined error: 0 (os error 0)
 2024-06-03T08:09:48.117Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.223Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.324Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.425Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.526Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.628Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.733Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.838Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.942Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pretty_env_logger-0.4.0/src/lib.rs:74:22
stack backtrace:
   0:        0x17c8bb580 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h819e9cbdf1a9e730
   1:        0x17c78b548 - core::fmt::write::ha5e9bf3131ecb7c0
   2:        0x17c8a10ac - std::io::Write::write_fmt::h414ce9994bf17404
   3:        0x17c8be880 - std::sys_common::backtrace::print::h8072db0bbd5bcc3d
   4:        0x17c8be4e0 - std::panicking::default_hook::{{closure}}::h2c85c5b0c2ede151
   5:        0x17c8bf3cc - std::panicking::rust_panic_with_hook::h023af7f90b47eb8b
   6:        0x17c8bef7c - std::panicking::begin_panic_handler::{{closure}}::h14283519edc1d634
   7:        0x17c8beeec - std::sys_common::backtrace::__rust_end_short_backtrace::hc366c0b0cef5b747
   8:        0x17c8beee0 - _rust_begin_unwind
   9:        0x17c8f9128 - core::panicking::panic_fmt::h324f50b29db90195
  10:        0x17c8f9374 - core::result::unwrap_failed::hf783e6a14bbaf60b
  11:        0x17c77e380 - _initialize_logging
  12:        0x1a4981050 - <unknown>
  13:        0x1a4989ae0 - <unknown>
  14:        0x17c189478 - _cdata_call
  15:        0x102289020 - __PyObject_MakeTpCall
  16:        0x10235ebdc - _call_function
  17:        0x10235bed4 - __PyEval_EvalFrameDefault
  18:        0x10235f898 - __PyEval_EvalCodeWithName
  19:        0x102289ba4 - __PyFunction_Vectorcall
  20:        0x10235eb3c - _call_function
  21:        0x10235bfbc - __PyEval_EvalFrameDefault
  22:        0x102289a04 - _function_code_fastcall
  23:        0x10235eb3c - _call_function
  24:        0x10235bed4 - __PyEval_EvalFrameDefault
  25:        0x10235f898 - __PyEval_EvalCodeWithName
  26:        0x102356150 - _PyEval_EvalCode
  27:        0x102353420 - _builtin_exec
  28:        0x1022cbab4 - _cfunction_vectorcall_FASTCALL
  29:        0x102289384 - _PyVectorcall_Call
  30:        0x10235c26c - __PyEval_EvalFrameDefault
  31:        0x10235f898 - __PyEval_EvalCodeWithName
  32:        0x102289ba4 - __PyFunction_Vectorcall
  33:        0x10235eb3c - _call_function
  34:        0x10235bed4 - __PyEval_EvalFrameDefault
  35:        0x102289a04 - _function_code_fastcall
  36:        0x10235eb3c - _call_function
  37:        0x10235beb0 - __PyEval_EvalFrameDefault
  38:        0x102289a04 - _function_code_fastcall
  39:        0x10235eb3c - _call_function
  40:        0x10235bf48 - __PyEval_EvalFrameDefault
  41:        0x102289a04 - _function_code_fastcall
  42:        0x10235eb3c - _call_function
  43:        0x10235bf48 - __PyEval_EvalFrameDefault
  44:        0x102289a04 - _function_code_fastcall
  45:        0x10235eb3c - _call_function
  46:        0x10235bf48 - __PyEval_EvalFrameDefault
  47:        0x10235f898 - __PyEval_EvalCodeWithName
  48:        0x102289ba4 - __PyFunction_Vectorcall
  49:        0x10235eb3c - _call_function
  50:        0x10235bed4 - __PyEval_EvalFrameDefault
  51:        0x10235f898 - __PyEval_EvalCodeWithName
  52:        0x102289ba4 - __PyFunction_Vectorcall
  53:        0x10235eb3c - _call_function
  54:        0x10235bed4 - __PyEval_EvalFrameDefault
  55:        0x102289a04 - _function_code_fastcall
  56:        0x10235eb3c - _call_function
  57:        0x10235bf48 - __PyEval_EvalFrameDefault
  58:        0x102289a04 - _function_code_fastcall
  59:        0x10235eb3c - _call_function
  60:        0x10235beb0 - __PyEval_EvalFrameDefault
  61:        0x10229b8f4 - _gen_send_ex
  62:        0x102daf01c - _task_step
  63:        0x102289020 - __PyObject_MakeTpCall
  64:        0x102378394 - _context_run
  65:        0x1022cbbdc - _cfunction_vectorcall_FASTCALL_KEYWORDS
  66:        0x102289384 - _PyVectorcall_Call
  67:        0x10235c26c - __PyEval_EvalFrameDefault
  68:        0x102289a04 - _function_code_fastcall
  69:        0x10235eb3c - _call_function
  70:        0x10235beb0 - __PyEval_EvalFrameDefault
  71:        0x102289a04 - _function_code_fastcall
  72:        0x10235eb3c - _call_function
  73:        0x10235beb0 - __PyEval_EvalFrameDefault
  74:        0x102289a04 - _function_code_fastcall
  75:        0x10235eb3c - _call_function
  76:        0x10235beb0 - __PyEval_EvalFrameDefault
  77:        0x102289a04 - _function_code_fastcall
  78:        0x10235eb3c - _call_function
  79:        0x10235beb0 - __PyEval_EvalFrameDefault
  80:        0x10235f898 - __PyEval_EvalCodeWithName
  81:        0x102289ba4 - __PyFunction_Vectorcall
  82:        0x10235eb3c - _call_function
  83:        0x10235beb0 - __PyEval_EvalFrameDefault
  84:        0x10235f898 - __PyEval_EvalCodeWithName
  85:        0x102289ba4 - __PyFunction_Vectorcall
  86:        0x10235eb3c - _call_function
  87:        0x10235bfbc - __PyEval_EvalFrameDefault
  88:        0x10235f898 - __PyEval_EvalCodeWithName
  89:        0x102356150 - _PyEval_EvalCode
  90:        0x10239ca78 - _PyRun_SimpleFileExFlags
  91:        0x1023bb078 - _Py_RunMain
  92:        0x1023bb508 - _pymain_main
  93:        0x1023bb5a8 - _Py_BytesMain
fatal runtime error: failed to initiate panic, error 5
[1]    66516 abort      RUST_BACKTRACE=full python app_fastapi.py