Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nested profiler signal handler #52765

Open
rmacnak-google opened this issue Jun 22, 2023 · 2 comments
Open

Nested profiler signal handler #52765

rmacnak-google opened this issue Jun 22, 2023 · 2 comments
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends.

Comments

@rmacnak-google
Copy link
Contributor 

../../runtime/vm/os_thread.h: 400: error: expected: !in_thread_interrupt_scope_

Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 3549023.3549754]
dart::Profiler::DumpStackTrace (sp=139691744026104, fp=139691744026416, pc=94545038494220, for_crash=<optimized out>) at out/DebugX64/../../runtime/vm/profiler.cc:454
454	  auto isolate = thread == nullptr ? nullptr : thread->isolate();
(rr) bt
#0  dart::Profiler::DumpStackTrace (sp=139691744026104, fp=139691744026416, pc=94545038494220, for_crash=<optimized out>)
    at out/DebugX64/../../runtime/vm/profiler.cc:454
#1  0x000055fcfb3021b4 in dart::Assert::Fail (this=<optimized out>, format=<optimized out>) at out/DebugX64/../../runtime/platform/assert.cc:53
#2  0x000055fcfb7ba9e8 in dart::ThreadInterruptScope::ThreadInterruptScope (this=<optimized out>) at ../../runtime/vm/os_thread.h:400
#3  dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler (signal=<optimized out>, info=<optimized out>, context_=<optimized out>)
    at out/DebugX64/../../runtime/vm/thread_interrupter_linux.cc:34
#4  <signal handler called>
#5  dart::Profiler::SampleThreadSingleFrame (thread=0x15505402fca0, sample=0x7d3142006710, pc=1879048197) at out/DebugX64/../../runtime/vm/profiler.cc:1295
#6  dart::Profiler::SampleThread (thread=0x15505402fca0, state=...) at out/DebugX64/../../runtime/vm/profiler.cc:1392
#7  0x000055fcfb7ba99c in dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler (signal=<optimized out>, info=<optimized out>, context_=<optimized out>)
    at out/DebugX64/../../runtime/vm/thread_interrupter_linux.cc:44
#8  <signal handler called>
#9  0x0000000070000005 in syscall_priv_traced ()
#10 0x00004e602e60a5f4 in _raw_syscall () at /build/rr-491BEd/rr-5.6.0/src/preload/raw_syscall.S:120
#11 0x00004e602e608257 in syscall_hook (call=0x7f0c84c7ffa0) at ./src/preload/syscallbuf.c:3989
#12 syscall_hook (call=0x7f0c84c7ffa0) at ./src/preload/syscallbuf.c:3933
#13 0x00004e602e6052e3 in _syscall_hook_trampoline () at /build/rr-491BEd/rr-5.6.0/src/preload/syscall_hook.S:308
#14 0x00004e602e60534d in __morestack () at /build/rr-491BEd/rr-5.6.0/src/preload/syscall_hook.S:443
#15 0x00004e602e6053cf in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-491BEd/rr-5.6.0/src/preload/syscall_hook.S:507
#16 0x00003a88143a9bb8 in __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=<optimized out>) at ./nptl/pthread_sigmask.c:45
#17 0x000055fcfb452d26 in dart::ThreadSignalBlocker::~ThreadSignalBlocker (this=0x248f4b1b1e10) at ../../runtime/platform/signal_blocker.h:46
#18 dart::bin::FDUtils::WriteToBlocking (fd=4, buffer=<optimized out>, count=24) at out/DebugX64/../../runtime/bin/fdutils_linux.cc:115
#19 0x000055fcfb2e7745 in dart::bin::EventHandlerImplementation::WakeupHandler (this=<optimized out>, id=<optimized out>, dart_port=<optimized out>, data=-1)
    at out/DebugX64/../../runtime/bin/eventhandler_linux.cc:186
#20 0x000055fcfb2e6ec0 in dart::bin::EventHandler::SendData (this=0x2, this@entry=0xffffffffffffffff, id=-1, dart_port=0, data=14)
    at ../../runtime/bin/eventhandler.h:609
#21 dart::bin::Builtin_EventHandler_SendData (args=args@entry=0x248f4b1b2040) at out/DebugX64/../../runtime/bin/eventhandler.cc:101
#22 0x000055fcfb5d42db in dart::NativeEntry::AutoScopeNativeCallWrapperNoStackCheck (args=0x248f4b1b2040, 
    func=0x55fcfb2e6e10 <dart::bin::Builtin_EventHandler_SendData(_Dart_NativeArguments*)>) at out/DebugX64/../../runtime/vm/native_entry.cc:217

@bkonyi The sigprof handler apparently does not block nested signals. This might be the underlying cause of the flaky crashes like #52213
@lrhn lrhn added the area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. label Jun 23, 2023
copybara-service bot pushed a commit that referenced this issue Jun 23, 2023
@rmacnak-google
[vm] Prevent nested handling of SIGPROF.
b9843b8 
TEST=ci
Bug: #52765
Change-Id: I14301d67ad245d3df96624155b383cb5f4c7f981
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/310975
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
@dcharkes
Copy link
Contributor

The mentioned test is now failing consistently instead of flakily:

https://dart-ci.appspot.com/log/pkg-linux-release/unittest-asserts-release-linux/23314/pkg/dds/test/get_cached_cpu_samples_test

--- Command "vm" (took 12.000738s):
DART_CONFIGURATION=ReleaseX64 CHROME_PATH=/b/s/w/ir/cache/builder/sdk/third_party/browsers/chrome/chrome/google-chrome out/ReleaseX64/dart-sdk/bin/dart --enable_asserts --sound-null-safety -Dtest_runner.configuration=unittest-asserts-release-linux --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.dart_tool/package_config.json /b/s/w/ir/cache/builder/sdk/pkg/dds/test/get_cached_cpu_samples_test.dart

exit code:
255

stdout:
00:00 �[32m+0�[0m: No UserTags to cache�[0m

TESTEE OUT: The Dart VM service is listening on http://127.0.0.1:34057/rUWq75T8mGw=/


TESTEE ERR: vm-service: isolate(1648990210023167) 'main' has no debugger attached and is paused at start.  Connect to the Dart VM service at http://127.0.0.1:34057/rUWq75T8mGw=/ to debug.


00:01 �[32m+1�[0m: Cache CPU samples for provided UserTag name�[0m

TESTEE OUT: The Dart VM service is listening on http://127.0.0.1:38011/Two-yilFyFw=/


TESTEE ERR: 
===== CRASH =====
si_signo=Segmentation fault(11), si_code=1, si_addr=0xb8
version=3.1.0-edge.b9843b876d5c4891f4e9cc6c721391f107ff1b12 (be) (Fri Jun 23 17:01:58 2023 +0000) on "linux_x64"
pid=234301, thread=234305, isolate_group=main(0x55ab5a78db10), isolate=(nil)((nil))
os=linux, arch=x64, comp=no, sim=no
isolate_instructions=55ab58dd7d60, vm_instructions=55ab58dd7d60
fp=7fdea68514c0, sp=7fdea6851470, pc=55ab59005030


TESTEE ERR:   pc 0x000055ab59005030 fp 0x00007fdea68514c0 dart::SampleBlockBuffer::BuildProcessedSampleBuffer(dart::SampleFilter*, dart::ProcessedSampleBuffer*)+0xb0
  pc 0x000055ab590093c0 fp 0x00007fdea6851500 out/ReleaseX64/dart-sdk/bin/dart+0x23753c0
  pc 0x000055ab59009342 fp 0x00007fdea68515a0 dart::Profile::Build(dart::Thread*, dart::SampleFilter*, dart::ProcessedSampleBufferBuilder*)+0x1f2
  pc 0x000055ab59006c32 fp 0x00007fdea6851cd0 dart::Profiler::ProcessCompletedBlocks(dart::Isolate*)+0xe2
  pc 0x000055ab59006edf fp 0x00007fdea6851cf0 out/ReleaseX64/dart-sdk/bin/dart+0x2372edf
  pc 0x000055ab58f494db fp 0x00007fdea6851d40 dart::IsolateGroup::ForEachIsolate(std::__2::function<void (dart::Isolate*)>, bool)+0x10b
  pc 0x000055ab58f514f6 fp 0x00007fdea6851d70 out/ReleaseX64/dart-sdk/bin/dart+0x22bd4f6
  pc 0x000055ab58f48c59 fp 0x00007fdea6851dc0 dart::IsolateGroup::ForEach(std::__2::function<void (dart::IsolateGroup*)>)+0xa9
  pc 0x000055ab58f4ff98 fp 0x00007fdea6851e00 dart::Isolate::VisitIsolates(dart::IsolateVisitor*)+0x38
  pc 0x000055ab59006b18 fp 0x00007fdea6851e30 dart::SampleBlockProcessor::ThreadMain(unsigned long)+0xa8
  pc 0x000055ab59000216 fp 0x00007fdea6851ef0 out/ReleaseX64/dart-sdk/bin/dart+0x236c216
-- End of DumpStackTrace


00:03 �[32m+1�[0m�[31m -1�[0m: Cache CPU samples for provided UserTag name �[1m�[31m[E]�[0m�[0m

  DartDevelopmentServiceException: Failed to start Dart Development Service

  package:dds/src/dds_impl.dart 136:7                  DartDevelopmentServiceImpl.startService
  ===== asynchronous gap ===========================
  package:dds/dds.dart 97:5                            DartDevelopmentService.startDartDevelopmentService
  ===== asynchronous gap ===========================
  pkg/dds/test/get_cached_cpu_samples_test.dart 71:13  main.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 215:9     Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 213:7     Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258:9      Invoker._waitForOutstandingCallbacks.<fn>
  

00:03 �[32m+1�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m



Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n unittest-asserts-release-linux pkg/dds/test/get_cached_cpu_samples_test

@derekxu16 derekxu16 reopened this Feb 2, 2024
@derekxu16
Copy link
Member

derekxu16 commented Feb 2, 2024 •
edited
Loading

I've managed to produce a similar stack trace using rr. It seems like there's some interaction with ~ThreadSignalBlocker that's causing problems, because in both traces, the segfault happens following ~ThreadSignalBlocker. Any thoughts at a glance @rmacnak-google or @bkonyi? If you'd like to investigate deeply I can try sharing my rr recording.

Thread 9 received signal SIGSEGV, Segmentation fault.
dart::Profiler::DumpStackTrace (sp=139986800205240, fp=139986800205552, pc=94420722290748, for_crash=<optimized out>) at out/DebugX64/../../runtime/vm/profiler.cc:506
506       auto isolate = thread == nullptr ? nullptr : thread->isolate();
(rr) bt
#0  dart::Profiler::DumpStackTrace (sp=139986800205240, fp=139986800205552, pc=94420722290748, for_crash=<optimized out>) at out/DebugX64/../../runtime/vm/profiler.cc:506
#1  0x000055e0095dab94 in dart::Assert::Fail (this=<optimized out>, format=<optimized out>) at out/DebugX64/../../runtime/platform/assert.cc:53
#2  0x000055e009a8edb8 in dart::ThreadInterruptScope::ThreadInterruptScope (this=<optimized out>) at ../../runtime/vm/os_thread.h:417
#3  dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler (signal=<optimized out>, info=<optimized out>, context_=<optimized out>)
    at out/DebugX64/../../runtime/vm/thread_interrupter_linux.cc:34
#4  <signal handler called>
#5  dart::VMTag::RuntimeEntryTagName (id=94420719576384) at out/DebugX64/../../runtime/vm/tags.cc:51
#6  0x000055e009a8107f in dart::VMTag::IsRuntimeEntryTag (id=94420719576384) at out/DebugX64/../../runtime/vm/tags.cc:48
#7  dart::VMTagCounters::Increment (this=0x55e0000286f0, tag=94420719576384) at out/DebugX64/../../runtime/vm/tags.cc:108
#8  0x000055e0099cb9fd in dart::Profiler::SampleThreadSingleFrame (thread=<optimized out>, sample=<optimized out>, pc=<optimized out>) at out/DebugX64/../../runtime/vm/profiler.cc:1364
#9  0x000055e0099cb9fd in dart::Profiler::SampleThread (thread=0x55e000003720, state=...)
#10 0x000055e009a8ed6c in dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler (signal=<optimized out>, info=<optimized out>, context_=<optimized out>)
    at out/DebugX64/../../runtime/vm/thread_interrupter_linux.cc:44
#11 <signal handler called>
#12 0x0000000070000005 in syscall_priv_traced ()
#13 0x00004a6839008525 in _raw_syscall () at /home/ubuntu/rr/src/preload/raw_syscall.S:120
#14 0x00004a6839005be9 in syscall_hook (call=0x7f51377fffa0) at /home/ubuntu/rr/src/preload/syscallbuf.c:4314
#15 0x00004a6839001353 in _syscall_hook_trampoline () at /home/ubuntu/rr/src/preload/syscall_hook.S:308
#16 0x00004a68390013bd in __morestack () at /home/ubuntu/rr/src/preload/syscall_hook.S:443
#17 0x00004a6839001456 in _syscall_hook_trampoline_89_c2_f7_da () at /home/ubuntu/rr/src/preload/syscall_hook.S:512
#18 0x000022263a61dfe8 in __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=<optimized out>) at ./nptl/pthread_sigmask.c:45
#19 0x000055e00973a2fb in dart::ThreadSignalBlocker::~ThreadSignalBlocker (this=0x7ebd0b815328) at ../../runtime/platform/signal_blocker.h:46
#20 dart::bin::File::Open (namespc=<optimized out>, name=<optimized out>, mode=dart::bin::File::kRead) at out/DebugX64/../../runtime/bin/file_linux.cc:252
#21 0x000055e009732588 in dart::bin::Builtin_File_Open (args=args@entry=0x7ebd0b815500) at out/DebugX64/../../runtime/bin/file.cc:100
#22 0x000055e0098b240b in dart::NativeEntry::AutoScopeNativeCallWrapperNoStackCheck (args=0x7ebd0b815500, func=0x55e009732540 <dart::bin::Builtin_File_Open(_Dart_NativeArguments*)>)
    at out/DebugX64/../../runtime/vm/native_entry.cc:219
#23 0x000055e0072832f4 in ?? ()
#24 0x000055e000003720 in ?? ()
#25 0x0000000000000003 in ?? ()
#26 0x00007ebd0b815560 in ?? ()
#27 0x00007ebd0b815548 in ?? ()
#28 0x000022263a52b081 in ?? ()
#29 0x000022263a53b911 in ?? ()
#30 0x00004a683928e1b1 in ?? ()
#31 0x00007ebd0b815580 in ?? ()
#32 0x000050ca39d81537 in ?? ()
#33 0x00004a6839288081 in ?? ()
#34 0x0000000000000000 in ?? ()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@derekxu16 @dcharkes @lrhn @rmacnak-google