Skip to content

Set benchmark-specific thread limits for the Spark local executor - #284

Merged
lbulej merged 1 commit into
masterfrom
topic/spark-thread-limits
Jul 4, 2021
Merged

lbulej merged 1 commit into
masterfrom
topic/spark-thread-limits

Conversation

@lbulej

@lbulej lbulej commented Jul 2, 2021

Copy link
Copy Markdown
Member

The limits reflect the recommendations from #260, with the limits for gauss-mix, chi-square and dec-tree bumped up by 2 threads (providing a bit of slack that does not help but should not hurt), resulting in the following defaults for the spark_thread_limit parameter:

  • 4 in chi-square and gauss-mix
  • 6 in dec-tree
  • 8 in movie-lens
  • 12 in als, als-ml, log-regression and page-rank
  • unlimited in naive-bayes

The limits reflect the recommendations in #260 (comment)
@lbulej
lbulej requested a review from ceresek July 2, 2021 11:11
@vhotspur

vhotspur commented Jul 2, 2021

Copy link
Copy Markdown
Member

Related to #283: perhaps there should be some hard minimum too?

@lbulej

lbulej commented Jul 2, 2021 •

Copy link
Copy Markdown
Member Author

Spark should be fine even with 1 CPU. On my machine (32G RAM, 4*2 CPUs), Spark benchmarks run (albeit slowly) under taskset -c 0, both on JDK8 and JDK11.

@lbulej

lbulej commented Jul 2, 2021 •

Copy link
Copy Markdown
Member Author

If the thread limits are fine, I'd suggest merging this, because the issue with #283 appears to be related to thread stack size. If we end up imposing a minimal number of threads to work around the thread stack size, it would be better to do it in another PR anyway.

@lbulej lbulej changed the title Set thread limits for the Spark local executor Set benchmark-specific thread limits for the Spark local executor Jul 2, 2021
@lbulej
lbulej merged commit bcecf23 into master Jul 4, 2021
@lbulej
lbulej deleted the topic/spark-thread-limits branch July 4, 2021 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants