Send analytics events with callstacks on panics and signals #1409

emilk · 2023-02-26T14:39:39Z

Pains have been taken to not reveal any sensitive part of the file path in the stacktrace.

The callstack only includes functions that is in the rerun source-code or in crates we call, so is not sensitive either. This will remain true as long as we only install these panic handlers in rerun binaries.

Even when we support user-plugins (with callbacks), users will have to write their own entry-point binaries to rerun, and should there be able to decide wgether or not to install our panic handlers, thus giving users the choice to not send callstacks that could include user code.

With this PR, we install crash handlers in rerun::run, which is called by the rerun binary and by python -m rerun.

We take care to trim the callstack from any calls before re_viewer::native::run_native_app to exclude any user-code for users who (forever reason) call rerun::run themselves.

When we have a crash running python -m rerun the bottom of the callstack looks like this:

 132: re_viewer::native::run_native_app
             at re_viewer/src/native.rs:32:5
 133: tokio::runtime::park::CachedParkThread::block_on
 134: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
 135: tokio::runtime::runtime::Runtime::block_on
 136: rerun_bindings::python_bridge::_::__pyfunction_main
 137: pyo3::impl_::trampoline::cfunction_with_keywords
 138: _cfunction_call_varargs
 139: __PyObject_MakeTpCall
 140: _call_function
 141: __PyEval_EvalFrameDefault
 142: _function_code_fastcall
 143: _call_function
 144: __PyEval_EvalFrameDefault
 145: __PyEval_EvalCodeWithName
 146: _PyEval_EvalCode
 147: _builtin_exec
 148: _cfunction_vectorcall_FASTCALL
 149: _call_function
 150: __PyEval_EvalFrameDefault

I don't know if anything sensitive can be garnered from these __PyEval things, but I've decided to trim it anyway by culling everything following run_native_app.

Here is an example of an anonymized callstack on a Python crash, copied straight from PostHog:

             at rerun/src/crash_handler.rs:102:25
   3: _OSAtomicTestAndClearBarrier
   4: re_viewer::app::debug_menu::{{closure}}
             at re_viewer/src/app.rs:1494:42
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
   5: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
   6: egui::ui::Ui::with_layout_dyn
             at egui-0.21.0/src/ui.rs:2012:21
   7: egui::ui::Ui::with_layout
             at egui-0.21.0/src/ui.rs:2003:9
      egui::menu::menu_ui::{{closure}}::{{closure}}
             at egui-0.21.0/src/menu.rs:158:17
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
   8: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
      egui::containers::frame::Frame::show_dyn
             at egui-0.21.0/src/containers/frame.rs:220:19
   9: egui::containers::frame::Frame::show
             at egui-0.21.0/src/containers/frame.rs:211:9
      egui::menu::menu_ui::{{closure}}
             at egui-0.21.0/src/menu.rs:153:9
      egui::containers::area::Area::show
             at egui-0.21.0/src/containers/area.rs:228:21
  10: egui::menu::menu_ui
             at egui-0.21.0/src/menu.rs:150:26
  11: egui::menu::MenuState::show
             at egui-0.21.0/src/menu.rs:578:9
      egui::menu::MenuState::show_submenu::{{closure}}
             at egui-0.21.0/src/menu.rs:588:34
      core::option::Option<T>::map
             at core/src/option.rs:925:29
      egui::menu::MenuState::show_submenu
             at egui-0.21.0/src/menu.rs:587:40
  12: egui::menu::SubMenu::show
             at egui-0.21.0/src/menu.rs:535:21
  13: egui::menu::submenu_button
             at egui-0.21.0/src/menu.rs:128:5
      egui::ui::Ui::menu_button
             at egui-0.21.0/src/ui.rs:2171:13
      re_viewer::app::debug_menu
             at re_viewer/src/app.rs:1475:5
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  15: egui::ui::Ui::with_layout_dyn
             at egui-0.21.0/src/ui.rs:2012:21
  16: egui::ui::Ui::with_layout
             at egui-0.21.0/src/ui.rs:2003:9
      egui::menu::menu_ui::{{closure}}::{{closure}}
             at egui-0.21.0/src/menu.rs:158:17
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  17: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
      egui::containers::frame::Frame::show_dyn
             at egui-0.21.0/src/containers/frame.rs:220:19
  18: egui::containers::frame::Frame::show
             at egui-0.21.0/src/containers/frame.rs:211:9
      egui::menu::menu_ui::{{closure}}
             at egui-0.21.0/src/menu.rs:153:9
      egui::containers::area::Area::show
             at egui-0.21.0/src/containers/area.rs:228:21
  19: egui::menu::menu_ui
             at egui-0.21.0/src/menu.rs:150:26
  20: egui::menu::MenuState::show
             at egui-0.21.0/src/menu.rs:578:9
      egui::menu::MenuState::show_submenu::{{closure}}
             at egui-0.21.0/src/menu.rs:588:34
      core::option::Option<T>::map
             at core/src/option.rs:925:29
      egui::menu::MenuState::show_submenu
             at egui-0.21.0/src/menu.rs:587:40
  21: egui::menu::SubMenu::show
             at egui-0.21.0/src/menu.rs:535:21
  22: egui::menu::submenu_button
             at egui-0.21.0/src/menu.rs:128:5
      egui::ui::Ui::menu_button
             at egui-0.21.0/src/ui.rs:2171:13
      re_viewer::app::rerun_menu_button_ui::{{closure}}
             at re_viewer/src/app.rs:1054:9
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  24: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  25: egui::ui::Ui::with_layout_dyn
             at egui-0.21.0/src/ui.rs:2012:21
  26: egui::ui::Ui::with_layout
             at egui-0.21.0/src/ui.rs:2003:9
      egui::menu::menu_ui::{{closure}}::{{closure}}
             at egui-0.21.0/src/menu.rs:158:17
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  27: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
      egui::containers::frame::Frame::show_dyn
             at egui-0.21.0/src/containers/frame.rs:220:19
  28: egui::containers::frame::Frame::show
             at egui-0.21.0/src/containers/frame.rs:211:9
      egui::menu::menu_ui::{{closure}}
             at egui-0.21.0/src/menu.rs:153:9
      egui::containers::area::Area::show
             at egui-0.21.0/src/containers/area.rs:228:21
  29: egui::menu::menu_ui
             at egui-0.21.0/src/menu.rs:150:26
  30: egui::menu::MenuState::show
             at egui-0.21.0/src/menu.rs:578:9
      egui::menu::MenuRoot::show
             at egui-0.21.0/src/menu.rs:295:17
  31: egui::menu::MenuRootManager::show
             at egui-0.21.0/src/menu.rs:244:51
  32: egui::menu::BarState::bar_menu
             at egui-0.21.0/src/menu.rs:49:9
      egui::menu::stationary_menu_image_impl
             at egui-0.21.0/src/menu.rs:208:17
  33: egui::menu::menu_image_button
             at egui-0.21.0/src/menu.rs:114:5
      egui::ui::Ui::menu_image_button
             at egui-0.21.0/src/ui.rs:2210:13
      re_viewer::app::rerun_menu_button_ui
             at re_viewer/src/app.rs:1010:5
      re_viewer::app::top_bar_ui
             at re_viewer/src/app.rs:1072:5
  34: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  35: egui::ui::Ui::allocate_ui_with_layout_dyn
             at egui-0.21.0/src/ui.rs:880:19
  36: egui::ui::Ui::horizontal_with_main_wrap_dyn
             at egui-0.21.0/src/ui.rs:1922:9
      egui::ui::Ui::horizontal
             at egui-0.21.0/src/ui.rs:1850:9
      egui::menu::bar
             at egui-0.21.0/src/menu.rs:79:5
  37: re_viewer::app::top_panel::{{closure}}
             at re_viewer/src/app.rs:981:29
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  38: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  39: egui::containers::panel::TopBottomPanel::show_inside_dyn::{{closure}}
             at egui-0.21.0/src/containers/panel.rs:728:13
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  40: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
      egui::containers::frame::Frame::show_dyn
             at egui-0.21.0/src/containers/frame.rs:220:19
  41: egui::containers::frame::Frame::show
             at egui-0.21.0/src/containers/frame.rs:211:9
      egui::containers::panel::TopBottomPanel::show_inside_dyn
             at egui-0.21.0/src/containers/panel.rs:725:30
  42: egui::containers::panel::TopBottomPanel::show_inside
             at egui-0.21.0/src/containers/panel.rs:649:9
      re_viewer::app::top_panel
             at re_viewer/src/app.rs:977:5
  43: <re_viewer::app::App as eframe::epi::App>::update::{{closure}}
             at re_viewer/src/app.rs:449:17
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  44: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
  45: egui::containers::panel::CentralPanel::show_inside_dyn::{{closure}}
             at egui-0.21.0/src/containers/panel.rs:1024:13
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at core/src/ops/function.rs:507:5
  46: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at alloc/src/boxed.rs:2000:9
      egui::containers::frame::Frame::show_dyn
             at egui-0.21.0/src/containers/frame.rs:220:19
  47: egui::containers::frame::Frame::show
             at egui-0.21.0/src/containers/frame.rs:211:9
      egui::containers::panel::CentralPanel::show_inside_dyn
             at egui-0.21.0/src/containers/panel.rs:1022:9
  48: egui::containers::panel::CentralPanel::show_dyn
             at egui-0.21.0/src/containers/panel.rs:1050:30
  49: egui::containers::panel::CentralPanel::show
             at egui-0.21.0/src/containers/panel.rs:1034:9
      <re_viewer::app::App as eframe::epi::App>::update
             at re_viewer/src/app.rs:442:9
  50: eframe::native::epi_integration::EpiIntegration::update::{{closure}}
             at eframe-0.21.3/src/native/epi_integration.rs:454:13
      egui::context::Context::run
             at egui-0.21.0/src/context.rs:286:9
  51: eframe::native::epi_integration::EpiIntegration::update
             at eframe-0.21.3/src/native/epi_integration.rs:452:27
  52: <eframe::native::run::wgpu_integration::WgpuWinitApp as eframe::native::run::WinitApp>::paint
             at eframe-0.21.3/src/native/run.rs:1223:21
  53: eframe::native::run::run_and_return::{{closure}}
             at eframe-0.21.3/src/native/run.rs:144:17
  54: <winit::platform_impl::platform::app_state::EventLoopHandler<T> as winit::platform_impl::platform::app_state::EventHandler>::handle_nonuser_event::{{closure}}
             at winit-0.28.1/src/platform_impl/macos/app_state.rs:96:17
      winit::platform_impl::platform::app_state::EventLoopHandler<T>::with_callback
             at winit-0.28.1/src/platform_impl/macos/app_state.rs:70:13
      <winit::platform_impl::platform::app_state::EventLoopHandler<T> as winit::platform_impl::platform::app_state::EventHandler>::handle_nonuser_event
             at winit-0.28.1/src/platform_impl/macos/app_state.rs:91:9
  55: winit::platform_impl::platform::app_state::Handler::handle_nonuser_event
             at winit-0.28.1/src/platform_impl/macos/app_state.rs:199:21
  56: winit::platform_impl::platform::app_state::AppState::cleared
             at winit-0.28.1/src/platform_impl/macos/app_state.rs:388:13
  57: winit::platform_impl::platform::observer::control_flow_end_handler::{{closure}}
             at winit-0.28.1/src/platform_impl/macos/observer.rs:79:21
      winit::platform_impl::platform::observer::control_flow_handler::{{closure}}
             at winit-0.28.1/src/platform_impl/macos/observer.rs:41:9
      std::panicking::try::do_call
             at std/src/panicking.rs:483:40
      std::panicking::try
             at std/src/panicking.rs:447:19
  58: std::panic::catch_unwind
             at std/src/panic.rs:137:14
      winit::platform_impl::platform::event_loop::stop_app_on_panic
             at winit-0.28.1/src/platform_impl/macos/event_loop.rs:245:11
      winit::platform_impl::platform::observer::control_flow_handler
             at winit-0.28.1/src/platform_impl/macos/observer.rs:39:5
      winit::platform_impl::platform::observer::control_flow_end_handler
             at winit-0.28.1/src/platform_impl/macos/observer.rs:74:9
  59: <unknown>
  60: <unknown>
  61: <unknown>
  62: <unknown>
  63: <unknown>
  64: <unknown>
  65: <unknown>
  66: <unknown>
  67: <unknown>
  68: <unknown>
  69: winit::platform_impl::platform::event_loop::EventLoop<T>::run_return::{{closure}}
             at winit-0.28.1/src/platform_impl/macos/event_loop.rs:220:22
      objc2::rc::autorelease::autoreleasepool
             at objc2-0.3.0-beta.3/src/rc/autorelease.rs:313:5
      winit::platform_impl::platform::event_loop::EventLoop<T>::run_return
             at winit-0.28.1/src/platform_impl/macos/event_loop.rs:211:25
  70: <winit::event_loop::EventLoop<T> as winit::platform::run_return::EventLoopExtRunReturn>::run_return
             at winit-0.28.1/src/platform/run_return.rs:51:9
      eframe::native::run::run_and_return
             at eframe-0.21.3/src/native/run.rs:124:5
  71: eframe::native::run::wgpu_integration::run_wgpu::{{closure}}
             at eframe-0.21.3/src/native/run.rs:1410:17
      eframe::native::run::with_event_loop::{{closure}}
             at eframe-0.21.3/src/native/run.rs:108:9
      std::thread::local::LocalKey<T>::try_with
             at std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at std/src/thread/local.rs:422:9
  72: eframe::native::run::with_event_loop
             at eframe-0.21.3/src/native/run.rs:101:5
      eframe::native::run::wgpu_integration::run_wgpu
             at eframe-0.21.3/src/native/run.rs:1407:13
  73: eframe::run_native
             at eframe-0.21.3/src/lib.rs:207:13
  74:

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I've added a line to CHANGELOG.md (if this is a big enough change to warrant it)

Wumpf

looks good to me!

crates/rerun/src/crash_handler.rs

emilk · 2023-02-27T15:31:48Z

I decided to add all the build-info as event properties too, so we can easily match a crash with the right version.

emilk · 2023-02-27T15:33:03Z

@Wumpf you may wanna take a second look at my post-review additions 😬

crates/rerun/src/crash_handler.rs

* Send analytics events with callstacks on panics and signals * Trim everything leading up to `run_native_app` * analytics is an acceptable label for CI * Merge fix * Include git branch and rust build date/time in default analytics * Send analytics last * Better function naming * Include BuildInfo in crash report analytics * Let's prefix the event names with what they are: crashes * Unify how build-info is added to Event:s

emilk added 2 commits February 26, 2023 15:12

Send analytics events with callstacks on panics and signals

952fb6a

Trim everything leading up to run_native_app

6ef822e

emilk added enhancement New feature or request 📊 analytics telemetry analytics labels Feb 26, 2023

emilk added 2 commits February 26, 2023 16:25

analytics is an acceptable label for CI

849cd8c

Merge branch 'main' into emilk/crash-analytics

12baafc

emilk added the 💣 crash crash, deadlock/freeze, do-no-start label Feb 26, 2023

emilk added 2 commits February 26, 2023 20:05

Merge branch 'main' into emilk/crash-analytics

012bead

Merge fix

4ef324b

Wumpf self-requested a review February 27, 2023 14:37

Include git branch and rust build date/time in default analytics

d19d624

Wumpf approved these changes Feb 27, 2023

View reviewed changes

crates/rerun/src/crash_handler.rs Show resolved Hide resolved

crates/rerun/src/crash_handler.rs Show resolved Hide resolved

crates/rerun/src/crash_handler.rs Outdated Show resolved Hide resolved

emilk added 3 commits February 27, 2023 16:12

Send analytics last

f47893a

Better function naming

f7b9baa

Include BuildInfo in crash report analytics

ffadf29

emilk requested a review from Wumpf February 27, 2023 15:29

Let's prefix the event names with what they are: crashes

622286a

Wumpf reviewed Feb 27, 2023

View reviewed changes

crates/rerun/src/crash_handler.rs Outdated Show resolved Hide resolved

emilk added 2 commits February 28, 2023 09:13

Unify how build-info is added to Event:s

5a06dcc

Merge branch 'main' into emilk/crash-analytics

484dc55

emilk merged commit 9dd0721 into main Feb 28, 2023

emilk deleted the emilk/crash-analytics branch February 28, 2023 08:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send analytics events with callstacks on panics and signals #1409

Send analytics events with callstacks on panics and signals #1409

emilk commented Feb 26, 2023 •

edited

Loading

Wumpf left a comment

emilk commented Feb 27, 2023

emilk commented Feb 27, 2023

Send analytics events with callstacks on panics and signals #1409

Send analytics events with callstacks on panics and signals #1409

Conversation

emilk commented Feb 26, 2023 • edited Loading

Checklist

Wumpf left a comment

Choose a reason for hiding this comment

emilk commented Feb 27, 2023

emilk commented Feb 27, 2023

emilk commented Feb 26, 2023 •

edited

Loading