Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock on isolate shutdown #26684

Closed
fsc8000 opened this issue Jun 10, 2016 · 1 comment
Closed

Deadlock on isolate shutdown #26684

fsc8000 opened this issue Jun 10, 2016 · 1 comment
Assignees
@fsc8000
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends.

Comments

@fsc8000
Copy link
Contributor

fsc8000 commented Jun 10, 2016

This occurs sometimes on the buildbot, causing flaky timeouts.

FAILED: none-vm release_x64 vm/dart/spawn_shutdown_test
Expected: Pass 
Actual: Timeout
CommandOutput[vm]:

stdout:
Starting 50 workers...
Waiting for 0:00:02.000000 before exit...


Command[vm]: DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --ignore-unrecognized-flags --package-root=xcodebuild/ReleaseX64/packages/ /b/build/slave/vm-mac-release-x64-be/build/sdk/runtime/tests/vm/dart/spawn_shutdown_test.dart
Took 0:01:00.017116

Short reproduction command (experimental):
    python tools/test.py -mrelease --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg -t60 vm/dart/spawn_shutdown_test
@fsc8000 fsc8000 added the area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. label Jun 10, 2016
@fsc8000 fsc8000 self-assigned this Jun 10, 2016
@fsc8000
Copy link
Contributor Author

fsc8000 commented Jun 10, 2016

Backtrace:

(lldb) thread backtrace all
* thread #1: tid = 0x1ee555, 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff95d2c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x000000010d68b162 dart`dart::Monitor::WaitMicros(this=<unavailable>, micros=<unavailable>) + 162 at os_thread_macos.cc:379
    frame #3: 0x000000010d5035ea dart`dart::Dart::Cleanup() [inlined] dart::MonitorLocker::Wait(millis=0) + 586 at lockers.h:169
    frame #4: 0x000000010d5035e2 dart`dart::Dart::Cleanup() [inlined] dart::Dart::WaitForIsolateShutdown() + 43 at dart.cc:319
    frame #5: 0x000000010d5035b7 dart`dart::Dart::Cleanup() + 535 at dart.cc:387
    frame #6: 0x000000010d3d0279 dart`::Dart_Cleanup() + 89 at dart_api_impl.cc:1179
    frame #7: 0x000000010d3bfee3 dart`dart::bin::main(argc=4, argv=0x00007fff52842a18) + 1827 at main.cc:1712
    frame #8: 0x000000010d3c0819 dart`main(argc=<unavailable>, argv=<unavailable>) + 9 at main.cc:1746
    frame #9: 0x00007fff90f815ad libdyld.dylib`start + 1
    frame #10: 0x00007fff90f815ad libdyld.dylib`start + 1

  thread #2: tid = 0x1ee556, 0x00007fff984d2eca libsystem_kernel.dylib`kevent + 10
    frame #0: 0x00007fff984d2eca libsystem_kernel.dylib`kevent + 10
    frame #1: 0x000000010d403f88 dart`dart::bin::EventHandlerImplementation::EventHandlerEntry(args=140495246390656) + 264 at eventhandler_macos.cc:461
    frame #2: 0x000000010d400dfe dart`dart::bin::ThreadStart(data_ptr=<unavailable>) + 30 at thread_macos.cc:83
    frame #3: 0x00007fff95d2b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #4: 0x00007fff95d2b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #5: 0x00007fff95d29351 libsystem_pthread.dylib`thread_start + 13

  thread #3: tid = 0x1ee569, 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff95d2c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x000000010d401692 dart`dart::bin::Monitor::WaitMicros(this=<unavailable>, micros=<unavailable>) + 162 at thread_macos.cc:301
    frame #3: 0x000000010d400b3a dart`dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) [inlined] dart::bin::MonitorLocker::Wait(millis=0) + 522 at lockers.h:44
    frame #4: 0x000000010d400b28 dart`dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) [inlined] dart::bin::Loader::BlockUntilComplete(this=<unavailable>) + 15 at loader.cc:210
    frame #5: 0x000000010d400b19 dart`dart::bin::Loader::LibraryTagHandler(tag=<unavailable>, library=<unavailable>, url=<unavailable>) + 489 at loader.cc:459
    frame #6: 0x000000010d3bf2f1 dart`dart::bin::CreateIsolateAndSetupHelper(script_uri="file:///Users/fschneider/dartgit/sdk/runtime/tests/vm/dart/spawn_shutdown_test.dart", main=<unavailable>, package_root="file:///Users/fschneider/dartgit/sdk/xcodebuild/ReleaseX64/packages/", packages_config=0x0000000000000000, flags=0x0000700001439e08, error=0x0000700001439e10, exit_code=0x0000700001439de4) + 1569 at main.cc:780
    frame #7: 0x000000010d3c03d1 dart`dart::bin::CreateIsolateAndSetup(script_uri=<unavailable>, main=<unavailable>, package_root=<unavailable>, package_config=<unavailable>, flags=<unavailable>, data=<unavailable>, error=<unavailable>) + 65 at main.cc:828
    frame #8: 0x000000010d443c9c dart`dart::SpawnIsolateTask::Run(this=0x00007fc79d303ca0) + 92 at isolate.cc:158
    frame #9: 0x000000010d7723c6 dart`dart::ThreadPool::Worker::Loop(this=0x00007fc79d001cc0) + 54 at thread_pool.cc:407
    frame #10: 0x000000010d77228c dart`dart::ThreadPool::Worker::Main(args=140495309249728) + 108 at thread_pool.cc:464
    frame #11: 0x000000010d68a6c7 dart`dart::ThreadStart(data_ptr=<unavailable>) + 71 at os_thread_macos.cc:97
    frame #12: 0x00007fff95d2b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #13: 0x00007fff95d2b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #14: 0x00007fff95d29351 libsystem_pthread.dylib`thread_start + 13

  thread #4: tid = 0x1ee57d, 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff95d2c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x000000010d401692 dart`dart::bin::Monitor::WaitMicros(this=<unavailable>, micros=<unavailable>) + 162 at thread_macos.cc:301
    frame #3: 0x000000010d400b3a dart`dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) [inlined] dart::bin::MonitorLocker::Wait(millis=0) + 522 at lockers.h:44
    frame #4: 0x000000010d400b28 dart`dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) [inlined] dart::bin::Loader::BlockUntilComplete(this=<unavailable>) + 15 at loader.cc:210
    frame #5: 0x000000010d400b19 dart`dart::bin::Loader::LibraryTagHandler(tag=<unavailable>, library=<unavailable>, url=<unavailable>) + 489 at loader.cc:459
    frame #6: 0x000000010d3bf2f1 dart`dart::bin::CreateIsolateAndSetupHelper(script_uri="file:///Users/fschneider/dartgit/sdk/runtime/tests/vm/dart/spawn_shutdown_test.dart", main=<unavailable>, package_root="file:///Users/fschneider/dartgit/sdk/xcodebuild/ReleaseX64/packages/", packages_config=0x0000000000000000, flags=0x0000700002875e08, error=0x0000700002875e10, exit_code=0x0000700002875de4) + 1569 at main.cc:780
    frame #7: 0x000000010d3c03d1 dart`dart::bin::CreateIsolateAndSetup(script_uri=<unavailable>, main=<unavailable>, package_root=<unavailable>, package_config=<unavailable>, flags=<unavailable>, data=<unavailable>, error=<unavailable>) + 65 at main.cc:828
    frame #8: 0x000000010d443c9c dart`dart::SpawnIsolateTask::Run(this=0x00007fc79d30fdb0) + 92 at isolate.cc:158
    frame #9: 0x000000010d7723c6 dart`dart::ThreadPool::Worker::Loop(this=0x00007fc79950bc20) + 54 at thread_pool.cc:407
    frame #10: 0x000000010d77228c dart`dart::ThreadPool::Worker::Main(args=140495247424544) + 108 at thread_pool.cc:464
    frame #11: 0x000000010d68a6c7 dart`dart::ThreadStart(data_ptr=<unavailable>) + 71 at os_thread_macos.cc:97
    frame #12: 0x00007fff95d2b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #13: 0x00007fff95d2b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #14: 0x00007fff95d29351 libsystem_pthread.dylib`thread_start + 13

  thread #5: tid = 0x1ee595, 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff95d2c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x000000010d68b162 dart`dart::Monitor::WaitMicros(this=<unavailable>, micros=<unavailable>) + 162 at os_thread_macos.cc:379
    frame #3: 0x000000010d5d4dda dart`dart::ShutdownIsolate(unsigned long) [inlined] dart::MonitorLocker::Wait(millis=0) + 74 at lockers.h:169
    frame #4: 0x000000010d5d4dd2 dart`dart::ShutdownIsolate(unsigned long) [inlined] dart::Isolate::WaitForOutstandingSpawns(this=<unavailable>) + 29 at isolate.cc:2467
    frame #5: 0x000000010d5d4db5 dart`dart::ShutdownIsolate(parameter=<unavailable>) + 37 at isolate.cc:1462
    frame #6: 0x000000010d7723c6 dart`dart::ThreadPool::Worker::Loop(this=0x00007fc79d207110) + 54 at thread_pool.cc:407
    frame #7: 0x000000010d77228c dart`dart::ThreadPool::Worker::Main(args=140495311368464) + 108 at thread_pool.cc:464
    frame #8: 0x000000010d68a6c7 dart`dart::ThreadStart(data_ptr=<unavailable>) + 71 at os_thread_macos.cc:97
    frame #9: 0x00007fff95d2b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #10: 0x00007fff95d2b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #11: 0x00007fff95d29351 libsystem_pthread.dylib`thread_start + 13

  thread #6: tid = 0x1ee5a5, 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff984d1db6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff95d2c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x000000010d68b162 dart`dart::Monitor::WaitMicros(this=<unavailable>, micros=<unavailable>) + 162 at os_thread_macos.cc:379
    frame #3: 0x000000010d5d4dda dart`dart::ShutdownIsolate(unsigned long) [inlined] dart::MonitorLocker::Wait(millis=0) + 74 at lockers.h:169
    frame #4: 0x000000010d5d4dd2 dart`dart::ShutdownIsolate(unsigned long) [inlined] dart::Isolate::WaitForOutstandingSpawns(this=<unavailable>) + 29 at isolate.cc:2467
    frame #5: 0x000000010d5d4db5 dart`dart::ShutdownIsolate(parameter=<unavailable>) + 37 at isolate.cc:1462
    frame #6: 0x000000010d7723c6 dart`dart::ThreadPool::Worker::Loop(this=0x00007fc79951ae80) + 54 at thread_pool.cc:407
    frame #7: 0x000000010d77228c dart`dart::ThreadPool::Worker::Main(args=140495247486592) + 108 at thread_pool.cc:464
    frame #8: 0x000000010d68a6c7 dart`dart::ThreadStart(data_ptr=<unavailable>) + 71 at os_thread_macos.cc:97
    frame #9: 0x00007fff95d2b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #10: 0x00007fff95d2b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #11: 0x00007fff95d29351 libsystem_pthread.dylib`thread_start + 13

fsc8000 added a commit that referenced this issue Jun 10, 2016
@fsc8000
Fix dead lock in isolate shutdown.
bbe812f 
The service isolate is shutdown after the application isolates, so that any blocking i/o requests by pending isolate spawn tasks can be handled.

BUG=#26684
R=zra@google.com

Review URL: https://codereview.chromium.org/2052343002 .
@fsc8000 fsc8000 closed this as completed Jun 10, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fsc8000 fsc8000

Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fsc8000