[SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S#45982
[SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S#45982beliefer wants to merge 6 commits intoapache:masterfrom
Conversation
docs/job-scheduling.md
Outdated
There was a problem hiding this comment.
This line change is irrelevant to K8s.
There was a problem hiding this comment.
Yes.
Because the link is not accurate even if it was not related to K8S.
Do you want me revert it?
There was a problem hiding this comment.
Yes, please handle this separately, @beliefer .
As you know, I'm the release manager of Apache Spark 3.4.3 (for next Monday).
I can backport this fix to branch-3.4 as a part of Apache Spark 3.4.3.
docs/job-scheduling.md
Outdated
There was a problem hiding this comment.
This looks wrong to me because there is actually nothing to follow in the future work section. May I ask what do you want to say here?
There was a problem hiding this comment.
How about In K8S mode, K8S doesn't support external shuffle service yet. ?
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Is this a required change, @beliefer?
I think we should add these document so as follows the other cluster mangers. |
beliefer
changed the title
|
I didn't even notice this page or section. The dropdown from the Navi Bar is enough for me. Do we support scheduling jobs across applications? It's odd to me. |
|
Nit: Use K8s instead of K8S, the former is the official abbreviation |
beliefer
force-pushed
the
SPARK-47795
branch
2 times, most recently
from
d6486d5 to
09ecc74
Compare
This section is about scheduling across applications. |
docs/job-scheduling.md
Outdated
There was a problem hiding this comment.
Given the content, ### Caveats section would be better than here.
beliefer
changed the title
Co-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @beliefer .
dongjoon-hyun
changed the title
docs/job-scheduling.md
Outdated
| ### Caveats | ||
|
|
||
| - In [standalone mode](spark-standalone.html), without explicitly setting `spark.executor.cores`, each executor will get all the available cores of a worker. In this case, when dynamic allocation enabled, spark will possibly acquire much more executors than expected. When you want to use dynamic allocation in [standalone mode](spark-standalone.html), you are recommended to explicitly set cores for each executor before the issue [SPARK-30299](https://issues.apache.org/jira/browse/SPARK-30299) got fixed. | ||
| - In [K8s mode](running-on-kubernetes.html), we can't using this feature by set `spark.shuffle.service.enabled` to `true` due to Spark on K8s doesn't support external shuffle service yet. |
There was a problem hiding this comment.
In K8s mode, we cannot use this feature by setting spark.shuffle.service.enabled to true because Spark on K8s does not yet support the external shuffle service.
| * **K8s:** The same as the situation with Yarn, please refer to the description of Yarn above. Furthermore, | ||
| Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.cores and | ||
| spark.kubernetes.executor.request.cores than spark.executor.cores. For more information, see the | ||
| [K8s Spark Properties](running-on-kubernetes.html#spark-properties). |
There was a problem hiding this comment.
spark.kubernetes.executor.limit.cores and
spark.kubernetes.executor.request.cores than spark.executor.cores -> spark.kubernetes.executor.limit.cores and
spark.kubernetes.executor.request.cores than spark.executor.cores
|
Merged to master Thank you @beliefer @dongjoon-hyun @bjornjorgensen |
|
@yaooqinn @dongjoon-hyun @bjornjorgensen Thank you! |
4 participants
What changes were proposed in this pull request?
This PR propose to supplement the doc of job schedule for K8S.
Why are the changes needed?
Spark document missing the description of job schedule for K8S.
Does this PR introduce any user-facing change?
'No'.
Just update document.
How was this patch tested?
Manual tests.
Was this patch authored or co-authored using generative AI tooling?
'No'.