[SPARK-21701][CORE] Enable RPC client to use SO_RCVBUF, SO_SNDBUF and SO_BACKLOG in SparkConf #18922
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…KLOG` in SparkConf. 2. Add Javadoc comment in MessageEncoder class. 3. Refine test case of asking non-existent endpoint in RPC test suite.
|
Can one of the admins verify this patch? |
|
@zsxwing Would you mind verifying the patch for me? I notice you have contributed to rpc module in spark. Many thanks! |
|
@neoremind can we separate different issues into different PRs? I saw you add several different fixes other than the PR title described. |
jerryshao
reviewed
Aug 16, 2017
| @@ -210,6 +210,18 @@ private TransportClient createClient(InetSocketAddress address) | |||
| .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, conf.connectionTimeoutMs()) | |||
| .option(ChannelOption.ALLOCATOR, pooledAllocator); | |||
|
|
|||
| if (conf.backLog() > 0) { | |||
| bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog()); | |||
There was a problem hiding this comment.
It is really used for Client bootstrap? AFAIK backlog is mainly used in server side to cache the incoming connections.
There was a problem hiding this comment.
I see your thoughts. I will eliminate the setting of the param.
|
@jerryshao Thanks for your reviewing! Per your advice, I have separated the issue into different PRs, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
TCP parameters like SO_RCVBUF, SO_SNDBUF and SO_BACKLOG can be set in
SparkConf, andorg.apache.spark.network.server.TransportServercan use those parameters to build server by leveraging netty. But forTransportClientFactory, there is no such way to set those parameters fromSparkConf. This could be inconsistent in server and client side when people set parameters inSparkConf. So this PR make RPC client to be enable to use those TCP parameters as well.Add some comments to
MessageDecoderclass to show how wire format looks like for a RPC message and draw a vivid graph to present to detailed wire protocol.How was this patch tested?
Existing tests and refine asking non-existent endpoint test case in
RpcEnvSuite.