Skip to content

[SPARK-18972][Core]Fix the netty thread names for RPC #16380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zsxwing wants to merge 1 commit into from
Closed

[SPARK-18972][Core]Fix the netty thread names for RPC #16380

zsxwing wants to merge 1 commit into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented Dec 22, 2016

What changes were proposed in this pull request?

Right now the name of threads created by Netty for Spark RPC are shuffle-client-** and shuffle-server-**. It's pretty confusing.

This PR just uses the module name in TransportConf to set the thread name. In addition, it also includes the following minor fixes:

  • TransportChannelHandler.channelActive and channelInactive should call the corresponding super methods.
  • Make ShuffleBlockFetcherIterator throw NoSuchElementException if it has no more elements. Otherwise, if the caller calls next without hasNext, it will just hang.

How was this patch tested?

Jenkins

mridulm
mridulm approved these changes Dec 22, 2016
View reviewed changes
Copy link
Contributor

@mridulm mridulm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; but might be a good idea to get someone else to also give it a look - been a while since I have played with netty :-)

Perhaps @rxin or someone else can also take a look ?

...on/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
}
super.channelRegistered(ctx);
super.channelActive(ctx);
}
Copy link
Contributor

@mridulm mridulm Dec 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch !

core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
if (!hasNext) {
throw new NoSuchElementException
}

Copy link
Contributor

@mridulm mridulm Dec 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of our iterators dont seem to enforce this invariant

Copy link
Member Author

@zsxwing zsxwing Dec 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true. But this hasNext is pretty cheap and it will throw an exception instead of hanging the thread, so it's better to have a check here.

Copy link
Member Author

@zsxwing zsxwing Dec 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @JoshRosen FYI, I have done the change we discussed here.

Copy link
Contributor

@mridulm mridulm Dec 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont get me wrong, I am proposing we do similar everywhere : not remove it from here !
This is the expectation from Iterator interface; which we do not honour

@rxin
Copy link
Contributor

rxin commented Dec 22, 2016

LGTM.

@SparkQA
Copy link

SparkQA commented Dec 22, 2016

Test build #70508 has finished for PR 16380 at commit 5b51040.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Dec 22, 2016

retest this please

@SparkQA
Copy link

SparkQA commented Dec 22, 2016

Test build #70510 has started for PR 16380 at commit 5b51040.

@SparkQA
Copy link

SparkQA commented Dec 22, 2016

Test build #3512 has finished for PR 16380 at commit 5b51040.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Dec 22, 2016

retest this please

@SparkQA
Copy link

SparkQA commented Dec 23, 2016

Test build #70527 has finished for PR 16380 at commit 5b51040.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Dec 23, 2016
@zsxwing
[SPARK-18972][CORE] Fix the netty thread names for RPC
2d72160 
## What changes were proposed in this pull request?

Right now the name of threads created by Netty for Spark RPC are `shuffle-client-**` and `shuffle-server-**`. It's pretty confusing.

This PR just uses the module name in TransportConf to set the thread name. In addition, it also includes the following minor fixes:

- TransportChannelHandler.channelActive and channelInactive should call the corresponding super methods.
- Make ShuffleBlockFetcherIterator throw NoSuchElementException if it has no more elements. Otherwise,  if the caller calls `next` without `hasNext`, it will just hang.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #16380 from zsxwing/SPARK-18972.

(cherry picked from commit f252cb5)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>
@zsxwing
Copy link
Member Author

zsxwing commented Dec 23, 2016

Thanks! Merged to master, 2.1 and 2.0.

@asfgit asfgit closed this in f252cb5 Dec 23, 2016
asfgit pushed a commit that referenced this pull request Dec 23, 2016
@zsxwing
[SPARK-18972][CORE] Fix the netty thread names for RPC
1857acc 
## What changes were proposed in this pull request?

Right now the name of threads created by Netty for Spark RPC are `shuffle-client-**` and `shuffle-server-**`. It's pretty confusing.

This PR just uses the module name in TransportConf to set the thread name. In addition, it also includes the following minor fixes:

- TransportChannelHandler.channelActive and channelInactive should call the corresponding super methods.
- Make ShuffleBlockFetcherIterator throw NoSuchElementException if it has no more elements. Otherwise,  if the caller calls `next` without `hasNext`, it will just hang.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #16380 from zsxwing/SPARK-18972.

(cherry picked from commit f252cb5)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Dec 24, 2016
@zsxwing @cmonkey
[SPARK-18972][CORE] Fix the netty thread names for RPC
ee4fc1b 
## What changes were proposed in this pull request?

Right now the name of threads created by Netty for Spark RPC are `shuffle-client-**` and `shuffle-server-**`. It's pretty confusing.

This PR just uses the module name in TransportConf to set the thread name. In addition, it also includes the following minor fixes:

- TransportChannelHandler.channelActive and channelInactive should call the corresponding super methods.
- Make ShuffleBlockFetcherIterator throw NoSuchElementException if it has no more elements. Otherwise,  if the caller calls `next` without `hasNext`, it will just hang.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes apache#16380 from zsxwing/SPARK-18972.
@zsxwing zsxwing deleted the SPARK-18972 branch January 13, 2017 07:02
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@zsxwing @uzadude
[SPARK-18972][CORE] Fix the netty thread names for RPC
1efa750 
## What changes were proposed in this pull request?

Right now the name of threads created by Netty for Spark RPC are `shuffle-client-**` and `shuffle-server-**`. It's pretty confusing.

This PR just uses the module name in TransportConf to set the thread name. In addition, it also includes the following minor fixes:

- TransportChannelHandler.channelActive and channelInactive should call the corresponding super methods.
- Make ShuffleBlockFetcherIterator throw NoSuchElementException if it has no more elements. Otherwise,  if the caller calls `next` without `hasNext`, it will just hang.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes apache#16380 from zsxwing/SPARK-18972.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mridulm mridulm mridulm approved these changes

@srowen srowen srowen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zsxwing @rxin @SparkQA @srowen @mridulm