Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-4370] [Core] Limit number of Netty cores based on executor size #3155

Closed
wants to merge 2 commits into from
Closed

[SPARK-4370] [Core] Limit number of Netty cores based on executor size #3155

wants to merge 2 commits into from

Conversation

aarondav
Copy link
Contributor

@aarondav aarondav commented Nov 7, 2014

No description provided.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23050 has started for PR 3155 at commit 5645c30.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23050 has finished for PR 3155 at commit 5645c30.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23050/
Test PASSed.

@aarondav aarondav changed the title [WIP] Allow disabling direct allocation in NettyBlockTransferService [SPARK-4370] [Core] Limit number of Netty cores based on executor size Nov 12, 2014
@aarondav
[SPARK-4370] [Core] Limit number of Netty cores based on executor size
4770f6e 
Right now, the NettyBlockTransferService uses the total number of cores on the system as the number of threads and buffer arenas to create. The latter is more troubling – this can lead to significant allocation of extra heap and direct memory in situations where executors are relatively small compared to the whole machine. For instance, on a machine with 32 cores, we will allocate (32 cores * 16MB per arena = 512MB) * 2 for client and server = 1GB direct and heap memory. This can be a huge overhead if you're only using, say, 8 of those cores.
@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23271 has started for PR 3155 at commit 4770f6e.

  • This patch merges cleanly.

@aarondav
Copy link
Contributor Author

@rxin This has been updated and should be merged into branch-1.2.0.

@tnachen Could you verify the correctness of getting the num cores from mesos? I don't have an easy way of testing there.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23272 has started for PR 3155 at commit 7045e77.

  • This patch merges cleanly.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23270/
Test FAILed.

@aarondav
Copy link
Contributor Author

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23274 has started for PR 3155 at commit 7045e77.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23271 has finished for PR 3155 at commit 4770f6e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23271/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23272 has finished for PR 3155 at commit 7045e77.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23272/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Nov 12, 2014

Test build #23274 has finished for PR 3155 at commit 7045e77.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class NettyBlockTransferService(conf: SparkConf, securityManager: SecurityManager, numCores: Int)

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23274/
Test PASSed.

@rxin
Copy link
Contributor

rxin commented Nov 12, 2014

The changes LGTM. @tnachen can you look at the Mesos related change?

@tnachen
Copy link
Contributor

tnachen commented Nov 13, 2014

@rxin @aarondav The change LGTM as well, +1

@rxin
Copy link
Contributor

rxin commented Nov 13, 2014

Merging in master & branch-1.2. Thanks!

@asfgit asfgit closed this in b9e1c2e Nov 13, 2014
asfgit pushed a commit that referenced this pull request Nov 13, 2014
@aarondav @rxin
[SPARK-4370] [Core] Limit number of Netty cores based on executor size
70efcd8 
Author: Aaron Davidson <aaron@databricks.com>

Closes #3155 from aarondav/conf and squashes the following commits:

7045e77 [Aaron Davidson] Add mesos comment
4770f6e [Aaron Davidson] [SPARK-4370] [Core] Limit number of Netty cores based on executor size

(cherry picked from commit b9e1c2e)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@aarondav @SparkQA @AmplabJenkins @rxin @tnachen