Profiler tries to acquire mutex in signal handler

[mkustermann]

From an internal crash:

F0124 20:17:02.002853  527579 mutex.cc:2080] RAW: Check waitp->thread->waitp == nullptr || waitp->thread->suppress_fatal_errors failed: detected illegal recursion into Mutex code
    @     0x556446335439  absl::raw_log_internal::RawLog()
    @     0x5564462ef943  absl::Mutex::LockSlowLoop()
    @     0x5564462eeb8b  absl::Mutex::LockSlowWithDeadline()
    @     0x5564462f177e  absl::Mutex::LockSlow()
    @     0x5564462edf7e  absl::Mutex::Lock()
    @     0x55644626b629  dart::SampleBlockBuffer::ReserveSampleImpl()
    @     0x55644626c3df  dart::Profiler::SampleThread()
    @     0x5564462d6c80  dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler()
    @     0x7fa2ffbb61c0  __restore_rt
    @     0x556446023e6c  AbslInternalPerThreadSemWait
    @     0x5564462f0e6c  absl::CondVar::WaitCommon()
    @     0x556446267aac  dart::Monitor::Wait()
    @     0x556446147d9e  dart::SafepointHandler::ExitSafepointUsingLock()
    @     0x5564461b7ffd  dart::MessageHandler::PauseAndHandleAllMessages()
    @     0x556446112e39  Dart_WaitForEvent
    @     0x5564460f0e49  dart::bin::Builtin_CLI_WaitForEvent()
    @     0x5564461cb4b8  dart::NativeEntry::AutoScopeNativeCallWrapperNoStackCheck()
    @     0x7fa2feb72ed3  _kDartVmSnapshotInstructions
    @     0x7fa2fecd4c69  _kDartIsolateSnapshotInstructions
    @     0x7fa2fecd4c2e  _kDartIsolateSnapshotInstructions
    @     0x7fa2fecd4b45  _kDartIsolateSnapshotInstructions
    @     0x7fa2fecd47a0  _kDartIsolateSnapshotInstructions

Signal handlers should only call functions that are async signal safe, see manpage

/cc @rmacnak-google @bkonyi

[rmacnak-google]

We have a longer list of restrictions documented on SampleThread:

  // SampleThread is called from inside the signal handler and hence it is very
  // critical that the implementation of SampleThread does not do any of the
  // following:
  //   * Accessing TLS -- Because on Windows and Fuchsia the callback will be
  //                      running in a different thread.
  //   * Allocating memory -- Because this takes locks which may already be
  //                          held, resulting in a dead lock.
  //   * Taking a lock -- See above.
  static void SampleThread(Thread* thread, const InterruptedThreadState& state);

[mkustermann]

Wasn't really involved in this, but I thought to recall that VM used to setup a preallocated ring-buffer where signal handlers can readily write into. Is this no longer the case?

@bkonyi would this be something you could look at?

[bkonyi]

I don't really have any cycles right now. Maybe @rmacnak-google can take a look?

[a-siva]

This locking code seems to have been added here https://dart-review.googlesource.com/c/sdk/+/206920

[jacob314]

This bug is slowing us down debugging a P0 issue in g3.

[rmacnak-google]

https://dart-review.googlesource.com/c/sdk/+/283703