[SPARK-30949][K8S][CORE] Decouple requests and parallelism on drivers in K8s #27695

onursatici · 2020-02-25T16:51:24Z

What changes were proposed in this pull request?

spark.driver.cores configuration is used to set the amount of parallelism in kubernetes cluster mode drivers. Previously the amount of parallelism in the drivers were the number of cores in the host when running on JDK 8u120 or older, or the maximum of driver containers resource requests and limits when running on JDK 8u121 or newer. This will enable users to specify spark.driver.cores to set parallelism, and specify spark.kubernetes.driver.requests.cores to limit the resource requests of the driver container, effectively decoupling the two

Why are the changes needed?

Drivers submitted in kubernetes cluster mode set the parallelism of various components like RpcEnv, MemoryManager, BlockManager from inferring the number of available cores by calling Runtime.getRuntime().availableProcessors(). By using this, spark applications running on JDK 8u120 or older incorrectly get the total number of cores in the host, ignoring the cgroup limits set by kubernetes. JDK 8u121 and newer runtimes do not have this problem.

Orthogonal to this, it is currently not possible to decouple resource limits on the driver container with the amount of parallelism of the various network and memory components listed above.

Does this PR introduce any user-facing change?

Yes. Previously the amount of parallelism in kubernetes cluster mode submitted drivers were the number of cores in the host when running on JDK 8u120 or older, or the maximum of driver containers resource requests and limits when running on JDK 8u121 or newer. Now the value of spark.driver.cores is used.

How was this patch tested?

happy to add tests if my proposal looks reasonable

dongjoon-hyun · 2020-02-25T22:07:23Z

ok to test

dongjoon-hyun

Hi, @onursatici . Thank you for making a PR, but the following claim seems to be outdated. Technically, it was fixed long time ago (on January 2017 at JDK 8u121).

By using this, spark applications running on java 8 or older incorrectly get the total number of cores in the host, ignoring the cgroup limits set by kubernetes. Java 9 and newer runtimes do not have this problem.

Please see https://bugs.openjdk.java.net/browse/JDK-8173345 for the detail.

Given the above, could you revise the PR description?

SparkQA · 2020-02-25T22:59:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23681/

SparkQA · 2020-02-25T23:32:36Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23681/

SparkQA · 2020-02-26T01:29:23Z

Test build #118933 has finished for PR 27695 at commit 2b3ad5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-26T02:02:49Z

Ping @holdenk since the flakiness of Test basic decommissioning is observed again.

- Test basic decommissioning *** FAILED ***
  The code passed to eventually never returned normally. Attempted 121 times over 2.01174358735 minutes. Last failure message: "++ id -u

dongjoon-hyun · 2020-02-29T02:25:34Z

Gentle ping, @onursatici .

holdenk · 2020-02-29T02:41:44Z

In my experience the K8s tests have all been flaky but I’ll dig into them & decom as well this coming week

jiangxb1987 · 2020-03-03T01:46:05Z

IIUC this issue also affects Standalone cluster mode?

onursatici · 2020-03-04T16:43:27Z

@dongjoon-hyun, what are the next steps? Does it look fine from your perspective?

dongjoon-hyun · 2020-03-04T20:39:50Z

@onursatici . One thing I'm thinking is the deprecation of 8u120 and the older versions at 3.0.0. Until now, 3.0.0-preview2 gave some early warning (not official deprecation yet) for 8u91 and the older versions like the following.

https://spark.apache.org/docs/3.0.0-preview2/

Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0

jiangxb1987 · 2020-03-17T17:57:44Z

Since this also affects Standalone cluster, I'd suggest we only exclude Mesos backend in the case match.

dongjoon-hyun · 2020-03-17T18:15:14Z

Hi, @jiangxb1987 . What do you mean by this? This PR or this old JDK bug?

Since this also affects Standalone cluster, I'd suggest we only exclude Mesos backend in the case match.

jiangxb1987 · 2020-03-17T18:17:31Z

I mean the JDK bug mentioned in this PR.

onursatici · 2020-03-25T14:38:45Z

@jiangxb1987 do you recommend to do that in this PR? I think changing the stand-alone driver core count behaviour would broaden the scope of this PR such that it might warrant a separate discussion.

Does this change make sense for k8s? Any blockers @dongjoon-hyun?

jiangxb1987 · 2020-03-25T17:33:35Z

It's fine if you want to focus on the k8s behaviour here, I can submit another PR to fix the Standalone backend after this is merged.

jiangxb1987

LGTM

jiangxb1987 · 2020-03-25T17:34:15Z

retest this please

SparkQA · 2020-03-25T18:18:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25078/

SparkQA · 2020-03-25T18:37:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25078/

SparkQA · 2020-03-25T19:51:58Z

Test build #120369 has finished for PR 27695 at commit 2b3ad5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-03-26T05:37:46Z

retest this please

SparkQA · 2020-03-26T06:31:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25107/

SparkQA · 2020-03-26T06:55:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25107/

SparkQA · 2020-03-26T07:05:03Z

Test build #120398 has finished for PR 27695 at commit 2b3ad5b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-03-26T17:36:44Z

retest this please

SparkQA · 2020-03-26T18:31:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25142/

SparkQA · 2020-03-26T18:50:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25142/

SparkQA · 2020-03-26T20:22:36Z

Test build #120434 has finished for PR 27695 at commit 2b3ad5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

onursatici · 2020-04-14T12:32:08Z

@dongjoon-hyun do you mind taking a look at this? I have revised the PR description

dongjoon-hyun

To merge this PR, it seems that we need to revise spark.kubernetes.driver.request.cores documentation together. The following is the current one. Although it's correct, I guess we need to mention the decoupled parallelism additionally due to spark.driver.cores after this PR.

https://github.com/apache/spark/blame/master/docs/running-on-kubernetes.md#L876

This takes precedence over <code>spark.driver.cores</code>
for specifying the driver pod cpu request if set.

dongjoon-hyun · 2020-04-15T03:46:35Z

@onursatici . I'm still not sure about this approach.

First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
Second, this original YARN code is made by the following PR for Netty. In K8s environment, the total number of available cores is the same with the requested amount.
- [SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations #21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)

The last possibility which I can guess is that you want to have bigger parallelism on the small container. Is that your case? Could you give us more concrete example where this PR is beneficial?

dongjoon-hyun · 2020-04-16T21:39:14Z

How do you think about the above comment, @onursatici ? I'm wondering your opinion.

dongjoon-hyun · 2020-04-18T17:07:53Z

Gentle ping, @onursatici .

onursatici · 2020-04-20T15:18:18Z

Hey @dongjoon-hyun , sorry for the late response. You are right that the feature I wan't with this PR is to mainly increase parallelism while keeping the cpu resource requests low in K8S.

spark.driver.cores controls the number of threads in various components, including some network components that are mainly blocked by I/O. The ability to increase this while keeping the cpu requests the same would allow users to better utilise their K8S clusters, especially if their workloads are I/O bound

dongjoon-hyun · 2020-04-21T03:02:57Z

Got it. Thanks, @onursatici . I'll revise the PR description a little and do the final review.

dongjoon-hyun

+1, LGTM. Thank you, @onursatici and @jiangxb1987 .
Merged to master for Apache Spark 3.1.0.

… in K8s ### What changes were proposed in this pull request? `spark.driver.cores` configuration is used to set the amount of parallelism in kubernetes cluster mode drivers. Previously the amount of parallelism in the drivers were the number of cores in the host when running on JDK 8u120 or older, or the maximum of driver containers resource requests and limits when running on [JDK 8u121 or newer](https://bugs.openjdk.java.net/browse/JDK-8173345). This will enable users to specify `spark.driver.cores` to set parallelism, and specify `spark.kubernetes.driver.requests.cores` to limit the resource requests of the driver container, effectively decoupling the two ### Why are the changes needed? Drivers submitted in kubernetes cluster mode set the parallelism of various components like `RpcEnv`, `MemoryManager`, `BlockManager` from inferring the number of available cores by calling `Runtime.getRuntime().availableProcessors()`. By using this, spark applications running on JDK 8u120 or older incorrectly get the total number of cores in the host, [ignoring the cgroup limits set by kubernetes](https://bugs.openjdk.java.net/browse/JDK-6515172). JDK 8u121 and newer runtimes do not have this problem. Orthogonal to this, it is currently not possible to decouple resource limits on the driver container with the amount of parallelism of the various network and memory components listed above. ### Does this PR introduce any user-facing change? Yes. Previously the amount of parallelism in kubernetes cluster mode submitted drivers were the number of cores in the host when running on JDK 8u120 or older, or the maximum of driver containers resource requests and limits when running on JDK 8u121 or newer. Now the value of `spark.driver.cores` is used. ### How was this patch tested? happy to add tests if my proposal looks reasonable Closes apache#27695 from onursatici/os/decouple-requests-and-parallelism. Authored-by: Onur Satici <onursatici@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

decouple requests and parallelism on kubernetes drivers

2b3ad5b

dongjoon-hyun added the KUBERNETES label Feb 25, 2020

dongjoon-hyun changed the title ~~[SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers~~ [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers Feb 25, 2020

dongjoon-hyun added the SPARK CORE label Feb 25, 2020

dongjoon-hyun requested changes Feb 25, 2020

View reviewed changes

dongjoon-hyun removed the SPARK CORE label Feb 28, 2020

jiangxb1987 approved these changes Mar 25, 2020

View reviewed changes

dongjoon-hyun reviewed Apr 15, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers~~ [SPARK-30949][K8S][CORE] Decouple requests and parallelism on kubernetes drivers Apr 21, 2020

dongjoon-hyun changed the title ~~[SPARK-30949][K8S][CORE] Decouple requests and parallelism on kubernetes drivers~~ [SPARK-30949][K8S][CORE] Decouple requests and parallelism on drivers in K8s Apr 21, 2020

dongjoon-hyun approved these changes Apr 21, 2020

View reviewed changes

dongjoon-hyun closed this in ad96510 Apr 21, 2020

onursatici mentioned this pull request Jun 25, 2020

[SPARK-30949][K8S][CORE] Decouple requests and parallelism on drivers… palantir/spark#698

Merged

[SPARK-30949][K8S][CORE] Decouple requests and parallelism on drivers in K8s #27695

[SPARK-30949][K8S][CORE] Decouple requests and parallelism on drivers in K8s #27695

Conversation

onursatici commented Feb 25, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Feb 25, 2020

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

SparkQA commented Feb 25, 2020

SparkQA commented Feb 25, 2020

SparkQA commented Feb 26, 2020

dongjoon-hyun commented Feb 26, 2020

dongjoon-hyun commented Feb 29, 2020

holdenk commented Feb 29, 2020

jiangxb1987 commented Mar 3, 2020

onursatici commented Mar 4, 2020

dongjoon-hyun commented Mar 4, 2020 • edited Loading

jiangxb1987 commented Mar 17, 2020

dongjoon-hyun commented Mar 17, 2020 • edited Loading

jiangxb1987 commented Mar 17, 2020

onursatici commented Mar 25, 2020

jiangxb1987 commented Mar 25, 2020

jiangxb1987 left a comment

Choose a reason for hiding this comment

jiangxb1987 commented Mar 25, 2020

SparkQA commented Mar 25, 2020

SparkQA commented Mar 25, 2020

SparkQA commented Mar 25, 2020

jiangxb1987 commented Mar 26, 2020

SparkQA commented Mar 26, 2020

SparkQA commented Mar 26, 2020

SparkQA commented Mar 26, 2020

jiangxb1987 commented Mar 26, 2020

SparkQA commented Mar 26, 2020

SparkQA commented Mar 26, 2020

SparkQA commented Mar 26, 2020

onursatici commented Apr 14, 2020

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 15, 2020 • edited Loading

dongjoon-hyun commented Apr 16, 2020

dongjoon-hyun commented Apr 18, 2020

onursatici commented Apr 20, 2020

dongjoon-hyun commented Apr 21, 2020 • edited Loading

dongjoon-hyun left a comment

Choose a reason for hiding this comment

onursatici commented Feb 25, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Mar 4, 2020 •

edited

Loading

dongjoon-hyun commented Mar 17, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Apr 15, 2020 •

edited

Loading

dongjoon-hyun commented Apr 21, 2020 •

edited

Loading