Netty server getting blocked under heavy read/writes #13305
-
In Uber, we are using Netty to handle the shuffle data from the Spark applications (the project is open source: RemoteShuffleService. We have a 300 node Netty cluster handling ~10-15PB, and 20000-30000 concurrent connections per server everyday. We use a 400 thread worker group and a separate boss group. At a very high level, server has an endpoint each for writing/reading data to/from disk. Data is streamed by the clients in chunks of ~4-5 MB We face intermittent timeout issues on the cluster especially when there are heavy read/writes. After investigation we had few questions:
Consider the case above. With some custom logging, I could see that both // code within the future and // code outside of the future get handled by the same EventLoop registered with Channel (again correct me if I'm wrong). // code outside of the future gets executed only after the future completes (there is no sendFileChannelFuture.await() or sync()). Is this expected? If it is, what's the advantage of using a Future here apart from bypassing the buffering.
Expected behaviorActual behaviorSteps to reproduceMinimal yet complete reproducer code (or URL to code)Netty version4.1.65.Final JVM version (e.g.
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 2 replies
-
These are a lot of different questions... I will try to answer all of them but please in the future create different discussion for each as it make it easer for people to find these etc.
|
Beta Was this translation helpful? Give feedback.
-
Thanks @normanmaurer for the answers. Just one more question, execution within the ctx.writeAndFlush() would be taken up by an EventLoop on which Channel was originally registered even if the ChannelHandler which invoked ctx.writeAndFlush() was run by a custom EventLoopGroup. How do we ensure that such executions be also taken up by a custom EventLoopGroup? |
Beta Was this translation helpful? Give feedback.
-
Awesome. Thanks for the detailed explanation! In majority of the cases we see that its the control requests which timeout on server end. By control requests here, I mean requests which do not do any disk I/O. This problem is also faced by Netty based server used by Apache Spark to handle shuffle data. Note that the netty server in Apache Spark shuffle service only handles data reads and no data writes (and yet people run into timeout issues). You can find discussion about it here: apache/spark#22173 This could be handled to some extent if there is a way to do a soft reservation in EventLoopGroup for certain kind of requests (control requests in our case). But since the I/O work is always handled by default EventLoop, such requests can always saturate the entire EventLoopGroop. Any way to tackle this? |
Beta Was this translation helpful? Give feedback.
-
Thanks everyone for the detailed explanation. Appreciate it :) |
Beta Was this translation helpful? Give feedback.
These are a lot of different questions... I will try to answer all of them but please in the future create different discussion for each as it make it easer for people to find these etc.
This is because it simplifies the whole threading model a lot. By handling everything on the same EventLoop there is no need for any extra synchronisation etc. Also it ensures everything is delivered in the correct order. This model is widely used by network frameworks.
I will be handled by the assigned EventLoop of the Channel and then handed over to the EventExecutor that handles the ChannelHandler
While the operation might execute directly there is no guarantee that this is the case. The only way…