Skip to content

🐞 Node.js v22.13.1 Deadlock in TLS operations when the V8 garbage collector runs concurrently with TLS buffer operations #58656

Open
@praveenin

Description

@praveenin

Version

v22.13.1

Platform

Linux 4.18.0-553.54.1.el8_10.x86_64

Subsystem

No response

What steps will reproduce the bug?

Encountered a deadlock in Node.js v22.13.1. The process appears to hang indefinitely with the main thread stuck inside __pthread_rwlock_wrlock. This appears to be triggered via V8 internals, particularly during v8::internal::MapUpdater::ReconfigureToDataField.

Backtrace:

* thread #1, name = 'node', stop reason = signal SIGSTOP
  * frame #0: 0x00007fdd55b17df5 libpthread.so.0`__GI___pthread_rwlock_wrlock [inlined] futex_abstimed_wait(private=0, abstime=0x0000000000000000, expected=2, futex_word=<unavailable>) at futex-internal.h:172:13
    frame #1: 0x00007fdd55b17dd4 libpthread.so.0`__GI___pthread_rwlock_wrlock [inlined] __pthread_rwlock_wrlock_full(abstime=0x0000000000000000, rwlock=0x0000000008133ce8) at pthread_rwlock_common.c:803:14
    frame #2: 0x00007fdd55b17c90 libpthread.so.0`__GI___pthread_rwlock_wrlock(rwlock=0x0000000008133ce8) at pthread_rwlock_wrlock.c:27:16
    frame #3: 0x0000000001736917 node`void heap::base::Stack::SetMarkerAndCallbackImpl<void v8::internal::LocalHeap::BlockWhileParked<v8::internal::ParkedSharedMutexGuardIf<(v8::base::MutexSharedType)0, (v8::base::NullBehavior)0>::ParkedSharedMutexGuardIf(v8::internal::LocalHeap*, v8::base::SharedMutex*, bool)::'lambda0'()>(v8::internal::ParkedSharedMutexGuardIf<(v8::base::MutexSharedType)0, (v8::base::NullBehavior)0>::ParkedSharedMutexGuardIf(v8::internal::LocalHeap*, v8::base::SharedMutex*, bool)::'lambda0'())::'lambda'()>(heap::base::Stack*, void*, void const*) + 71
    frame #4: 0x0000000001ce63d1 node`PushAllRegistersAndIterateStack + 27
    frame #5: 0x000000000173b7b6 node`v8::internal::MapUpdater::ReconfigureToDataField(v8::internal::InternalIndex, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::Handle<v8::internal::FieldType>) + 790
    frame #6: 0x000000000173c1e0 node`v8::internal::(anonymous namespace)::UpdateDescriptorForValue(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::InternalIndex, v8::internal::PropertyConstness, v8::internal::Handle<v8::internal::Object>) + 208
    frame #7: 0x0000000001731576 node`v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::StoreOrigin) + 134
    frame #8: 0x000000000174ce6a node`v8::internal::Object::TransitionAndWriteDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) + 106
    frame #9: 0x00000000018bfb45 node`v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) + 469
    frame #10: 0x00000000018c146b node`v8::internal::Runtime_SetKeyedProperty(int, unsigned long*, v8::internal::Isolate*) + 59
    frame #11: 0x00007fdd4ee6c476
    frame #12: 0x00007fdd2fc6df55
    frame #13: 0x00007fdd4ee0c9c3
    frame #14: 0x00007fdd4eeeb275
    frame #15: 0x00007fdd4edfc919
    frame #16: 0x00007fdd4edcb403
    frame #17: 0x00000000013c3615 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 853
    frame #18: 0x00000000013c484a node`v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) + 138
    frame #19: 0x00000000013f6f09 node`v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 217
    frame #20: 0x00000000013f72c3 node`v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) + 67
    frame #21: 0x00007fdd4edcf745
    frame #22: 0x00007fdd2f1d6a5c
    frame #23: 0x00007fdd4edcb4dc
    frame #24: 0x00007fdd4edcb203
    frame #25: 0x00000000013c33f3 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 307
    frame #26: 0x00000000013c4365 node`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 85
    frame #27: 0x0000000001274856 node`v8::Object::CallAsFunction(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 390
    frame #28: 0x0000000000e9156f node`node::InternalCallbackScope::Close() + 687
    frame #29: 0x0000000000e9193b node`node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) + 667
    frame #30: 0x0000000000eab372 node`node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) + 194
    frame #31: 0x00000000010e625b node`node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) + 171
    frame #32: 0x00000000010e6626 node`node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) + 502
    frame #33: 0x00000000011b791b node`node::crypto::TLSWrap::ClearOut() + 299
    frame #34: 0x00000000011b8e10 node`node::crypto::TLSWrap::OnStreamRead(long, uv_buf_t const&) + 160
    frame #35: 0x00000000010ee0cf node`node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) + 143
    frame #36: 0x00000000010ee4fa node`node::LibuvStreamWrap::ReadStart()::'lambda0'(uv_stream_s*, long, uv_buf_t const*)::_FUN(uv_stream_s*, long, uv_buf_t const*) + 90
    frame #37: 0x0000000001d59e09 node`uv__read(stream=0x00000000155f2920) at stream.c:1148:7
    frame #38: 0x0000000001d5a200 node`uv__stream_io(loop=<unavailable>, w=0x00000000155f2920, events=1) at stream.c:1208:5
    frame #39: 0x0000000001d623f4 node`uv__io_poll(loop=0x0000000006725420, timeout=<unavailable>) at linux.c:1564:11
    frame #40: 0x0000000001d4d5a7 node`uv_run(loop=0x0000000006725420, mode=UV_RUN_DEFAULT) at core.c:458:5
    frame #41: 0x0000000000e926d6 node`node::SpinEventLoopInternal(node::Environment*) + 342
    frame #42: 0x0000000000fe7d51 node`node::NodeMainInstance::Run() + 257
    frame #43: 0x0000000000f49dcb node`node::Start(int, char**) + 1003
    frame #44: 0x00007fdd55b03c40 libc.so.6
    frame #45: 0x00007fdd559740cb libc.so.6`__libc_start_main(main=(node`main), argc=24, argv=0x00007ffe73b0b2f8, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffe73b0b2e8) at libc-start.c:308:16
    frame #46: 0x0000000000e8e58e node`_start + 46
  thread #2, stop reason = signal SIGSTOP
    frame #0: 0x00007fdd55a43d56 libc.so.6`__GI_epoll_pwait(epfd=9, events=0x00007fdd55948ce0, maxevents=1024, timeout=-1, set=0x0000000000000000) at epoll_pwait.c:42:10
    frame #1: 0x0000000001d622e2 node`uv__io_poll(loop=0x000000000811a488, timeout=-1) at linux.c:1464:12
    frame #2: 0x0000000001d4d5a7 node`uv_run(loop=0x000000000811a488, mode=UV_RUN_DEFAULT) at core.c:458:5
    frame #3: 0x0000000001021480 node`node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start()::'lambda'(void*)::_FUN(void*) + 128
    frame #4: 0x00007fdd55b12f96 libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:486:8
    frame #5: 0x00007fdd55a43bef libc.so.6`__clone at clone.S:95
  thread #3, stop reason = signal SIGSTOP
    frame #0: 0x00007fdd55b178e0 libpthread.so.0`__GI___pthread_rwlock_rdlock [inlined] futex_abstimed_wait(private=0, abstime=0x0000000000000000, expected=3, futex_word=<unavailable>) at futex-internal.h:172:13
    frame #1: 0x00000000027f5119 node`void heap::base::Stack::SetMarkerForBackgroundThreadAndCallbackImpl<void v8::internal::LocalHeap::BlockWhileParked<v8::internal::ParkedSharedMutexGuardIf<(v8::base::MutexSharedType)1, (v8::base::NullBehavior)0>::ParkedSharedMutexGuardIf(v8::internal::LocalHeap*, v8::base::SharedMutex*, bool)::'lambda'()>(v8::internal::ParkedSharedMutexGuardIf<(v8::base::MutexSharedType)1, (v8::base::NullBehavior)0>::ParkedSharedMutexGuardIf(v8::internal::LocalHeap*, v8::base::SharedMutex*, bool)::'lambda'())::'lambda'()>(heap::base::Stack*, void*, void const*) + 137
    frame #2: 0x0000000001ce63d1 node`PushAllRegistersAndIterateStack + 27
    frame #3: 0x000000000286a38a node`v8::internal::compiler::AccessInfoFactory::ComputePropertyAccessInfo(v8::internal::compiler::MapRef, v8::internal::compiler::NameRef, v8::internal::compiler::AccessMode) const + 1290
    frame #4: 0x000000000280b18a node`v8::internal::compiler::JSHeapBroker::GetPropertyAccessInfo(v8::internal::compiler::MapRef, v8::internal::compiler::NameRef, v8::internal::compiler::AccessMode) + 906
    frame #5: 0x0000000002bf36db node`v8::internal::compiler::JSNativeContextSpecialization::ReduceNamedAccess(v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::NamedAccessFeedback const&, v8::internal::compiler::AccessMode, v8::internal::compiler::Node*) + 891
    frame #6: 0x0000000002bfb095 node`v8::internal::compiler::JSNativeContextSpecialization::ReduceJSLoadNamed(v8::internal::compiler::Node*) + 389
    frame #7: 0x00000000028951eb node`v8::internal::compiler::Reducer::Reduce(v8::internal::compiler::Node*, v8::internal::compiler::ObserveNodeManager*) + 27
    frame #8: 0x00000000028eb26f node`v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce(v8::internal::compiler::Node*) + 63
    frame #9: 0x00000000028952c1 node`v8::internal::compiler::GraphReducer::Reduce(v8::internal::compiler::Node*) + 145
    frame #10: 0x0000000002897074 node`v8::internal::compiler::GraphReducer::ReduceTop() + 436
    frame #11: 0x0000000002897499 node`v8::internal::compiler::GraphReducer::ReduceNode(v8::internal::compiler::Node*) + 121
    frame #12: 0x0000000002903194 node`v8::internal::compiler::PipelineImpl::CreateGraph() + 1348
    frame #13: 0x000000000290a34e node`v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl(v8::internal::RuntimeCallStats*, v8::internal::LocalIsolate*) + 158
    frame #14: 0x0000000001308e2b node`v8::internal::OptimizedCompilationJob::ExecuteJob(v8::internal::RuntimeCallStats*, v8::internal::LocalIsolate*) + 43
    frame #15: 0x00000000013482a1 node`v8::internal::OptimizingCompileDispatcher::CompileNext(v8::internal::TurbofanCompilationJob*, v8::internal::LocalIsolate*) + 49
    frame #16: 0x000000000134863f node`v8::internal::OptimizingCompileDispatcher::CompileTask::Run(v8::JobDelegate*) + 255
    frame #17: 0x000000000224b548 node`v8::platform::DefaultJobWorker::Run() + 136
    frame #18: 0x000000000101c3ec node`node::(anonymous namespace)::PlatformWorkerThread(void*) + 428
    frame #19: 0x00007fdd55b12f96 libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:486:8
    frame #20: 0x00007fdd55a43bef libc.so.6`__clone at clone.S:95
  thread #4, stop reason = signal SIGSTOP
    (same as thread #3)
  thread #5, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #6, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #7, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #8, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #9, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #10, stop reason = signal SIGSTOP
    (same as thread #2)
  thread #11, stop reason = signal SIGSTOP
    (same as thread #2)

How often does it reproduce? Is there a required condition?

Not known

What is the expected behavior? Why is that the expected behavior?

Nodejs process shouldn't hang

What do you see instead?

Nodejs process hangs

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs more infoIssues without a valid reproduction.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions