Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Occasional deadlock exceptions #3737

Closed
zhang0125 opened this issue Apr 12, 2021 · 1 comment
Closed

Occasional deadlock exceptions #3737

zhang0125 opened this issue Apr 12, 2021 · 1 comment
Labels
type:bug

Comments

@zhang0125
Copy link
Contributor

zhang0125 commented Apr 12, 2021 •
edited

1. What did you do?

I want to run FullNode.jar.

2. What did you expect to see?

Synchronize block continuously.

3. What did you see instead?

Block synchronization may stop working due to some deadlock, this problem is not 100% reproducible, but it is already very painful even if it happens occasionally.
Here are the relevant logs:

Java stack information for the threads listed above:
===================================================
"TronJClientWorker-28":
        at org.tron.common.overlay.server.SyncPool.onDisconnect(SyncPool.java:153)
        - waiting to lock <0x0000000384159de0> (a org.tron.common.overlay.server.SyncPool)
        at org.tron.common.overlay.server.ChannelManager.notifyDisconnect(ChannelManager.java:113)
        at org.tron.common.overlay.server.TronChannelInitializer.lambda$initChannel$0(TronChannelInitializer.java:50)
        at org.tron.common.overlay.server.TronChannelInitializer$$Lambda$400/1959924813.operationComplete(Unknown Source)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:945)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:901)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
        at org.tron.common.overlay.server.MessageQueue.fastSend(MessageQueue.java:90)
        at org.tron.core.net.peer.PeerConnection.fastSend(PeerConnection.java:113)
        at org.tron.core.net.service.AdvService$InvSender.lambda$null$4(AdvService.java:336)
        at org.tron.core.net.service.AdvService$InvSender$$Lambda$559/119921742.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.tron.core.net.service.AdvService$InvSender.lambda$sendFetch$5(AdvService.java:333)
        at org.tron.core.net.service.AdvService$InvSender$$Lambda$558/2114110900.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.tron.core.net.service.AdvService$InvSender.sendFetch(AdvService.java:333)
        at org.tron.core.net.service.AdvService.consumerInvToFetch(AdvService.java:253)
        - locked <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.service.AdvService.addInv(AdvService.java:133)
        - locked <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.messagehandler.InventoryMsgHandler.processMessage(InventoryMsgHandler.java:43)
        at org.tron.core.net.TronNetService.onMessage(TronNetService.java:99)
       ......
"nioEventLoopGroup-6-15":
        at org.tron.core.net.service.AdvService.consumerInvToFetch(AdvService.java:225)
        - waiting to lock <0x0000000383b7df58> (a org.tron.core.net.service.AdvService)
        at org.tron.core.net.service.AdvService.onDisconnect(AdvService.java:220)
        at org.tron.core.net.peer.PeerConnection.onDisconnect(PeerConnection.java:135)
        at org.tron.common.overlay.server.SyncPool.onDisconnect(SyncPool.java:161)
        - locked <0x0000000384159de0> (a org.tron.common.overlay.server.SyncPool)
        at org.tron.common.overlay.server.ChannelManager.notifyDisconnect(ChannelManager.java:113)
        at org.tron.common.overlay.server.TronChannelInitializer.lambda$initChannel$0(TronChannelInitializer.java:50)
        at org.tron.common.overlay.server.TronChannelInitializer$$Lambda$400/1959924813.operationComplete(Unknown Source)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1376)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465)
        at org.tron.common.overlay.server.Channel.close(Channel.java:159)
        at org.tron.common.overlay.server.P2pHandler.channelRead0(P2pHandler.java:77)
        at org.tron.common.overlay.server.P2pHandler.channelRead0(P2pHandler.java:20)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
@Benson0224 Benson0224 added the topic: Performance GC, OOM, CPU high label Apr 12, 2021
@Benson0224 Benson0224 added type:bug and removed topic: Performance GC, OOM, CPU high labels Apr 21, 2021
@Benson0224
Copy link
Contributor

We will close this issue Since it is already fixed. , please feel free to reopen it if you still see it, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zhang0125 @Benson0224