Skip to content

Conversation

@Eistern
Copy link

@Eistern Eistern commented Mar 27, 2025

This PR replaces synchronized blocks in the YDB SDK with ReentrantLock implementations to resolve deadlocks that occur when using AsyncReader with Java 23 virtual threads. Synchronized blocks inadvertently pin virtual threads to carrier threads, leading to thread starvation in high-concurrency scenarios.

This change unblocks adoption of virtual threads in YDB applications while maintaining backward compatibility.

Example stacktrace of the pinned virtual thread

#329476554 "reader-session-8/f998jjgudtv9jk4kf359-executor-worker-33912448" virtual
      java.base/jdk.internal.misc.Unsafe.park(Native Method)
      java.base/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:680)
      java.base/java.lang.VirtualThread.park(VirtualThread.java:612)
      java.base/java.lang.System$2.parkVirtualThread(System.java:2735)
      java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
      java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:219)
      java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
      java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
      java.base/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:154)
      java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:323)
      io.netty.buffer.PoolSubpage.lock(PoolSubpage.java:294)
      io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:160)
      io.netty.buffer.PoolArena.allocate(PoolArena.java:135)
      io.netty.buffer.PoolArena.allocate(PoolArena.java:127)
      io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
      io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
      io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
      io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
      io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:226)
      io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:172)
      io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:143)
      io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:66)
      io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
      io.grpc.internal.RetriableStream.sendMessage(RetriableStream.java:575)
      io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:522)
      io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:510)
      io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
      io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
      [...]
      io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
      tech.ydb.core.impl.call.ReadWriteStreamCall.sendNext(ReadWriteStreamCall.java:95) // the thread is pinned here
      tech.ydb.topic.impl.SessionBase.send(SessionBase.java:80)
      tech.ydb.topic.read.impl.ReaderImpl$ReadSessionImpl.sendReadRequest(ReaderImpl.java:269)
      tech.ydb.topic.read.impl.ReaderImpl$ReadSessionImpl.lambda$onReadResponse$6(ReaderImpl.java:472)
      [...]
      tech.ydb.topic.read.impl.Batch.complete(Batch.java:32)
      tech.ydb.topic.read.impl.PartitionSessionImpl.lambda$sendDataToReadersIfNeeded$7(PartitionSessionImpl.java:337)
      [...]
      java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
      java.base/java.lang.VirtualThread.run(VirtualThread.java:329)

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 20.68966% with 46 lines in your changes missing coverage. Please review.

Project coverage is 58.79%. Comparing base (291f913) to head (19ce14a).

Files with missing lines Patch % Lines
...a/tech/ydb/core/impl/call/ReadWriteStreamCall.java 0.00% 22 Missing ⚠️
...n/java/tech/ydb/core/impl/call/ReadStreamCall.java 0.00% 20 Missing ⚠️
...h/ydb/test/integration/docker/GrpcProxyServer.java 75.00% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master     #381      +/-   ##
============================================
+ Coverage     58.74%   58.79%   +0.04%     
- Complexity     2166     2167       +1     
============================================
  Files           320      320              
  Lines         12894    12900       +6     
  Branches       1265     1265              
============================================
+ Hits           7575     7584       +9     
+ Misses         4708     4705       -3     
  Partials        611      611              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Eistern Eistern marked this pull request as ready for review March 28, 2025 05:18
@alex268 alex268 merged commit 52d188e into ydb-platform:master Mar 28, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants