Skip to content

Occasional deadlock exceptions #3737

Closed
Closed
@zhang0125

Description

@zhang0125

1. What did you do?

I want to run FullNode.jar.

2. What did you expect to see?

Synchronize block continuously.

3. What did you see instead?

Block synchronization may stop working due to some deadlock, this problem is not 100% reproducible, but it is already very painful even if it happens occasionally.
Here are the relevant logs:

Java stack information for the threads listed above:
===================================================
"TronJClientWorker-28":
        at org.tron.common.overlay.server.SyncPool.onDisconnect(SyncPool.java:153)
        - waiting to lock <0x0000000384159de0> (a org.tron.common.overlay.server.SyncPool)
        at org.tron.common.overlay.server.ChannelManager.notifyDisconnect(ChannelManager.java:113)
        at org.tron.common.overlay.server.TronChannelInitializer.lambda$initChannel$0(TronChannelInitializer.java:50)
        at org.tron.common.overlay.server.TronChannelInitializer$$Lambda$400/1959924813.operationComplete(Unknown Source)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:945)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:901)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
        at org.tron.common.overlay.server.MessageQueue.fastSend(MessageQueue.java:90)
        at org.tron.core.net.peer.PeerConnection.fastSend(PeerConnection.java:113)
        at org.tron.core.net.service.AdvService$InvSender.lambda$null$4(AdvService.java:336)
        at org.tron.core.net.service.AdvService$InvSender$$Lambda$559/119921742.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.tron.core.net.service.AdvService$InvSender.lambda$sendFetch$5(AdvService.java:333)
        at org.tron.core.net.service.AdvService$InvSender$$Lambda$558/2114110900.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.tron.core.net.service.AdvService$InvSender.sendFetch(AdvService.java:333)
        at org.tron.core.net.service.AdvService.consumerInvToFetch(AdvService.java:253)
        - locked <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.service.AdvService.addInv(AdvService.java:133)
        - locked <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.messagehandler.InventoryMsgHandler.processMessage(InventoryMsgHandler.java:43)
        at org.tron.core.net.TronNetService.onMessage(TronNetService.java:99)
       ......
"nioEventLoopGroup-6-15":
        at org.tron.core.net.service.AdvService.consumerInvToFetch(AdvService.java:225)
        - waiting to lock <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.service.AdvService.onDisconnect(AdvService.java:220)
        at org.tron.core.net.peer.PeerConnection.onDisconnect(PeerConnection.java:135)
        at org.tron.common.overlay.server.SyncPool.onDisconnect(SyncPool.java:161)
        - locked <0x0000000384159de0> (a org.tron.common.overlay.server.SyncPool)
        at org.tron.common.overlay.server.ChannelManager.notifyDisconnect(ChannelManager.java:113)
        at org.tron.common.overlay.server.TronChannelInitializer.lambda$initChannel$0(TronChannelInitializer.java:50)
        at org.tron.common.overlay.server.TronChannelInitializer$$Lambda$400/1959924813.operationComplete(Unknown Source)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1376)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465)
        at org.tron.common.overlay.server.Channel.close(Channel.java:159)
        at org.tron.common.overlay.server.P2pHandler.channelRead0(P2pHandler.java:77)
        at org.tron.common.overlay.server.P2pHandler.channelRead0(P2pHandler.java:20)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions