Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSAN failure due to data race between FunctionDeserializationCluster::PostLoad and Function::EntryPointOf #49394

Closed
sstrickl opened this issue Jul 4, 2022 · 0 comments
Closed

TSAN failure due to data race between FunctionDeserializationCluster::PostLoad and Function::EntryPointOf #49394

sstrickl opened this issue Jul 4, 2022 · 0 comments
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening

Comments

@sstrickl
Copy link
Contributor

sstrickl commented Jul 4, 2022

Example warning when running python3 tools/test.py -n dartkp-tsan-linux-release-x64 vm/dart_2/isolates/fast_object_copy_test/3:

WARNING: ThreadSanitizer: data race (pid=1923295)
  Write of size 8 at 0x7ff2a0f0e698 by main thread:
    #0 dart::FunctionDeserializationCluster::PostLoad(dart::Deserializer*, dart::Array const&, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:1246:45 (dart_precompiled_runtime+0x845a03)
    #1 dart::Deserializer::Deserialize(dart::DeserializationRoots*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:8471:21 (dart_precompiled_runtime+0x83ebb7)
    #2 dart::FullSnapshotReader::ReadProgramSnapshot() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:8903:16 (dart_precompiled_runtime+0x83f7d0)
    #3 dart::Dart::InitIsolateFromSnapshot(dart::Thread*, dart::Isolate*, unsigned char const*, unsigned char const*, unsigned char const*, long) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart.cc:873:47 (dart_precompiled_runtime+0x869d7d)
    #4 dart::Dart::InitializeIsolate(unsigned char const*, unsigned char const*, unsigned char const*, long, dart::IsolateGroup*, void*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart.cc:994:9 (dart_precompiled_runtime+0x86a31b)
    #5 dart::CreateIsolate(dart::IsolateGroup*, bool, char const*, void*, char**) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart_api_impl.cc:1297:12 (dart_precompiled_runtime+0xeb7c31)
    #6 Dart_CreateIsolateGroup ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart_api_impl.cc:1394:26 (dart_precompiled_runtime+0xeb800a)
    #7 dart::bin::CreateIsolateGroupAndSetupHelper(bool, char const*, char const*, char const*, Dart_IsolateFlags*, void*, char**, int*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:819:13 (dart_precompiled_runtime+0x668c23)
    #8 dart::bin::RunMainIsolate(char const*, char const*, dart::bin::CommandLineOptions*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1022:26 (dart_precompiled_runtime+0x6687d7)
    #9 dart::bin::main(int, char**) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1403:9 (dart_precompiled_runtime+0x669942)
    #10 main ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1442:3 (dart_precompiled_runtime+0x66a36e)

  Previous read of size 8 at 0x7ff2a0f0e698 by thread T4 (mutexes: write M0):
    #0 dart::Function::EntryPointOf(dart::FunctionPtr) ../../out/ReleaseTSANX64/../../runtime/vm/object.h:2899:31 (dart_precompiled_runtime+0x89dd5c)
    #1 dart::IsolateGroup::ScheduleThreadLocked(dart::MonitorLocker*, dart::Thread*, bool, bool, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/isolate.cc:621:7 (dart_precompiled_runtime+0x89dd5c)
    #2 dart::IsolateGroup::ScheduleThread(bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/isolate.cc:714:10 (dart_precompiled_runtime+0x89e093)
    #3 dart::Thread::EnterIsolateGroupAsHelper(dart::IsolateGroup*, dart::Thread::TaskKind, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/thread.cc:356:35 (dart_precompiled_runtime+0xa6c1ea)
    #4 dart::ConcurrentMarkTask::Run() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/marker.cc:790:19 (dart_precompiled_runtime+0xaa3b68)
    #5 dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/thread_pool.cc:158:15 (dart_precompiled_runtime+0xa7305c)
    #6 dart::ThreadPool::Worker::Main(unsigned long) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/thread_pool.cc:330:9 (dart_precompiled_runtime+0xa73b35)
    #7 dart::ThreadStart(void*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/os_thread_linux.cc:154:5 (dart_precompiled_runtime+0x9bcd7d)

  Mutex M0 (0x7b1c00000310) created at:
    #0 pthread_mutex_init ../staging/llvm_build/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/../staging/llvm_build/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1316:3 (dart_precompiled_runtime+0x5e2d3f)
    #1 dart::Monitor::Monitor() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/os_thread_linux.cc:372:12 (dart_precompiled_runtime+0x9bd3e6)
    #2 dart::ThreadRegistry::ThreadRegistry() ../../out/ReleaseTSANX64/../../runtime/vm/thread_registry.h:25:22 (dart_precompiled_runtime+0x89bb73)
    #3 dart::IsolateGroup::IsolateGroup(std::__2::shared_ptr<dart::IsolateGroupSource>, void*, dart::ObjectStore*, Dart_IsolateFlags) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/isolate.cc:349:28 (dart_precompiled_runtime+0x89bb73)
    #4 dart::IsolateGroup::IsolateGroup(std::__2::shared_ptr<dart::IsolateGroupSource>, void*, Dart_IsolateFlags) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/isolate.cc:409:7 (dart_precompiled_runtime+0x89c49d)
    #5 Dart_CreateIsolateGroup ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart_api_impl.cc:1390:20 (dart_precompiled_runtime+0xeb7fd1)
    #6 dart::bin::CreateIsolateGroupAndSetupHelper(bool, char const*, char const*, char const*, Dart_IsolateFlags*, void*, char**, int*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:819:13 (dart_precompiled_runtime+0x668c23)
    #7 dart::bin::RunMainIsolate(char const*, char const*, dart::bin::CommandLineOptions*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1022:26 (dart_precompiled_runtime+0x6687d7)
    #8 dart::bin::main(int, char**) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1403:9 (dart_precompiled_runtime+0x669942)
    #9 main ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1442:3 (dart_precompiled_runtime+0x66a36e)

  Thread T4 'DartWorker' (tid=1923300, running) created by main thread at:
    #0 pthread_create ../staging/llvm_build/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/../staging/llvm_build/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022:3 (dart_precompiled_runtime+0x5e156d)
    #1 dart::OSThread::Start(char const*, void (*)(unsigned long), unsigned long) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/os_thread_linux.cc:173:12 (dart_precompiled_runtime+0x9bcc1c)
    #2 dart::ThreadPool::Worker::StartThread() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/thread_pool.cc:296:16 (dart_precompiled_runtime+0xa72744)
    #3 dart::ThreadPool::RunImpl(std::__2::unique_ptr<dart::ThreadPool::Task, std::__2::default_delete<dart::ThreadPool::Task>>) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/thread_pool.cc:94:17 (dart_precompiled_runtime+0xa72744)
    #4 bool dart::ThreadPool::Run<dart::ConcurrentMarkTask, dart::GCMarker*, dart::IsolateGroup* const&, dart::PageSpace*&, dart::MarkingVisitorBase<true>*&>(dart::GCMarker*&&, dart::IsolateGroup* const&, dart::PageSpace*&, dart::MarkingVisitorBase<true>*&) ../../out/ReleaseTSANX64/../../runtime/vm/thread_pool.h:46:12 (dart_precompiled_runtime+0xa9f400)
    #5 dart::GCMarker::StartConcurrentMark(dart::PageSpace*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/marker.cc:920:42 (dart_precompiled_runtime+0xa9f400)
    #6 dart::PageSpace::CollectGarbageHelper(dart::Thread*, bool, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/pages.cc:1203:14 (dart_precompiled_runtime+0xaad545)
    #7 dart::PageSpace::CollectGarbage(dart::Thread*, bool, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/pages.cc:1147:3 (dart_precompiled_runtime+0xaacf16)
    #8 dart::Heap::StartConcurrentMarking(dart::Thread*, dart::GCReason) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/heap.cc:620:14 (dart_precompiled_runtime+0xa99eb4)
    #9 dart::Heap::CheckConcurrentMarking(dart::Thread*, dart::GCReason, long) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/heap.cc:605:9 (dart_precompiled_runtime+0xa98d9f)
    #10 dart::PageSpace::TryAllocateInFreshPage(long, dart::FreeList*, dart::OldPage::PageType, dart::PageSpace::GrowthPolicy, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/pages.cc:476:14 (dart_precompiled_runtime+0xaa857a)
    #11 dart::PageSpace::TryAllocateInternal(long, dart::FreeList*, dart::OldPage::PageType, dart::PageSpace::GrowthPolicy, bool, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/pages.cc:561:16 (dart_precompiled_runtime+0xaa8a2b)
    #12 dart::PageSpace::TryAllocate(long, dart::OldPage::PageType, dart::PageSpace::GrowthPolicy) ../../out/ReleaseTSANX64/../../runtime/vm/heap/pages.h:336:12 (dart_precompiled_runtime+0xa9811c)
    #13 dart::Heap::AllocateOld(dart::Thread*, long, dart::OldPage::PageType) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/heap/heap.cc:98:29 (dart_precompiled_runtime+0xa9811c)
    #14 dart::Heap::Allocate(dart::Thread*, long, dart::Heap::Space) ../../out/ReleaseTSANX64/../../runtime/vm/heap/heap.h (dart_precompiled_runtime+0x8f5746)
    #15 dart::Object::Allocate(long, long, dart::Heap::Space, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/object.cc:2712:25 (dart_precompiled_runtime+0x8f5746)
    #16 dart::Array::NewUninitialized(long, long, dart::Heap::Space) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/object.cc:24366:9 (dart_precompiled_runtime+0x8f7f01)
    #17 dart::Array::NewUninitialized(long, dart::Heap::Space) ../../out/ReleaseTSANX64/../../runtime/vm/object.h:10453:12 (dart_precompiled_runtime+0x8f7f01)
    #18 dart::Array::Grow(dart::Array const&, long, dart::Heap::Space) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/object.cc:24445:27 (dart_precompiled_runtime+0x8f7f01)
    #19 dart::GrowableObjectArray::Grow(long, dart::Heap::Space) const ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/object.cc:24589:21 (dart_precompiled_runtime+0x8fdb0a)
    #20 dart::GrowableObjectArray::Add(dart::Object const&, dart::Heap::Space) const ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/object.cc:24577:5 (dart_precompiled_runtime+0x8fdb0a)
    #21 dart::Deserializer::EndInstructions() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:8335:12 (dart_precompiled_runtime+0x83e29f)
    #22 dart::CodeDeserializationCluster::PostLoad(dart::Deserializer*, dart::Array const&, bool) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:2389:8 (dart_precompiled_runtime+0x8476db)
    #23 dart::Deserializer::Deserialize(dart::DeserializationRoots*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:8471:21 (dart_precompiled_runtime+0x83ebb7)
    #24 dart::FullSnapshotReader::ReadProgramSnapshot() ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/app_snapshot.cc:8903:16 (dart_precompiled_runtime+0x83f7d0)
    #25 dart::Dart::InitIsolateFromSnapshot(dart::Thread*, dart::Isolate*, unsigned char const*, unsigned char const*, unsigned char const*, long) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart.cc:873:47 (dart_precompiled_runtime+0x869d7d)
    #26 dart::Dart::InitializeIsolate(unsigned char const*, unsigned char const*, unsigned char const*, long, dart::IsolateGroup*, void*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart.cc:994:9 (dart_precompiled_runtime+0x86a31b)
    #27 dart::CreateIsolate(dart::IsolateGroup*, bool, char const*, void*, char**) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart_api_impl.cc:1297:12 (dart_precompiled_runtime+0xeb7c31)
    #28 Dart_CreateIsolateGroup ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/vm/dart_api_impl.cc:1394:26 (dart_precompiled_runtime+0xeb800a)
    #29 dart::bin::CreateIsolateGroupAndSetupHelper(bool, char const*, char const*, char const*, Dart_IsolateFlags*, void*, char**, int*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:819:13 (dart_precompiled_runtime+0x668c23)
    #30 dart::bin::RunMainIsolate(char const*, char const*, dart::bin::CommandLineOptions*) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1022:26 (dart_precompiled_runtime+0x6687d7)
    #31 dart::bin::main(int, char**) ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1403:9 (dart_precompiled_runtime+0x669942)
    #32 main ../../out/ReleaseTSANX64/../../out/ReleaseTSANX64/../../runtime/bin/main.cc:1442:3 (dart_precompiled_runtime+0x66a36e)

Looking through the logs of the last non-purple run on the TSAN trybot, it looks like this is most if not all of the failures being seen there:

The following tests began failing:

standalone_2/io/socket_ipv6_test/03 broke (Pass -> Crash, expected Pass)
standalone_2/io/socket_simple_bind_test/0 broke (Pass -> Crash, expected Pass)
vm/dart_2/isolates/fast_object_copy2_test/2 broke (Pass -> Crash, expected Pass)
vm/dart_2/isolates/fast_object_copy_test/3 broke (Pass -> Crash, expected Pass)
vm/dart_2/isolates/long_regexp_process_oob_messages_test broke (Pass -> Crash, expected Pass)
vm/dart_2/isolates/spawn_generic_function_test broke (Pass -> Crash, expected Pass)
vm/dart_2/null_float64x2_simd_ops_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress29846_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress38979_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress40189_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_34755_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_35887_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_42065_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_42202_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_42418_1_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_42418_2_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_42799_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_b_216834909_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_flutter76919_test broke (Pass -> Crash, expected Pass)
vm/dart_2/regress_flutter_64526_test broke (Pass -> Crash, expected Pass)
vm/dart_2/scavenger_abort_2_test broke (Pass -> Crash, expected Pass)
vm/dart_2/sdk_hash_test broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/13 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/14 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/17 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/18 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/19 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/2 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/5 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/6 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/8 broke (Pass -> Crash, expected Pass)
vm/dart_2/splay_test/9 broke (Pass -> Crash, expected Pass)
vm/dart_2/split_aot_kernel_generation2_test broke (Pass -> RuntimeError, expected Pass)
vm/dart_2/split_aot_kernel_generation_test broke (Pass -> RuntimeError, expected Pass)
vm/dart_2/split_literals_test broke (Pass -> Crash, expected Pass)
vm/dart_2/switchable_call_single_target_regression_42517_test broke (Pass -> Crash, expected Pass)
vm/dart_2/truncating_ints_test broke (Pass -> Crash, expected Pass)
vm/dart_2/unboxed_fields_type_args_test broke (Pass -> Crash, expected Pass)
vm/dart_2/unboxed_param_tear_off_test broke (Pass -> Crash, expected Pass)
vm/dart_2/use_dwarf_stack_traces_flag_test broke (Pass -> RuntimeError, expected Pass)

/cc @rmacnak-google
@sstrickl sstrickl added area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening labels Jul 4, 2022
copybara-service bot pushed a commit that referenced this issue Jul 6, 2022
@rmacnak-google
[vm] Don't read function entry points when initializing Threads that …
1044c64 
…don't execute Dart code.

Such helper threads may start concurrently with PostLoad phase of deserialization, during which the function entry points are being initialized.

TEST=tsan
Bug: #49394
Change-Id: I4cad3ebc0d54364d508ba8af8091efebbb6ece3a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/250761
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sstrickl @rmacnak-google