-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-48210][DOC]Modify the description of whether dynamic partition… #46496
base: master
Are you sure you want to change the base?
Conversation
…ing is enabled in the “ Stage Level Scheduling Overview”
@cloud-fan hi, Can you help me review this PR? |
cc @mridulm and @tgravescs |
sorry this description is very hard to read. I think you are trying to say that stage level scheduling isn't supported on k8s and yarn when dynamic allocation is disabled? When you say dynamic partitions do you mean dynamic allocation of executors? That is not true we added support - https://issues.apache.org/jira/browse/SPARK-45495 The description as is:
Is correct. When dynamic allocation is off, you can specify a different resource profile that changes the task requiresments but uses the existing executors. |
@tgravescs Thank you for your reply. Can you help me check my configuration, code, and error messages returned. Is there a problem with my understanding? My set is: My code is: val rdd = sc.range(0, 9) Return info: ` ` ‘ Is there a problem with my understanding? |
what version of Spark are you using? That issue I pointed to was fixed in 4.0.0 and 3.5.1. The check and error that is on the main branch should read:
I'm guessing you are using a version of Spark before that change. |
@tgravescs |
…ing is enabled in the “ Stage Level Scheduling Overview”
What changes were proposed in this pull request?
“ Stage Level Scheduling Overview ” in running-on-yarn and running-on-kubernetes
The description of dynamic partitioning is inconsistent with the code implementation verification.
In running-on-yarn
'
When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup.
'
But the implementation is:
Class:ResourceProfileManager
Fuc:isSupported
private[spark] def isSupported(rp: ResourceProfile): Boolean = {
assert(master != null)
if (rp.isInstanceOf[TaskResourceProfile] && !dynamicEnabled) {
if ((notRunningUnitTests || testExceptionThrown) &&
!(isStandaloneOrLocalCluster || isYarn || isK8s))
{ throw new SparkException("TaskResourceProfiles are only supported for Standalone, " + "Yarn and Kubernetes cluster for now when dynamic allocation is disabled.") }
}
The judgment of this code is that it does not support TaskResourceProfile in Yarn and k8s when dynamic partitioning is closed.
The description in the document does not match, so the document needs to be modified.
Why are the changes needed?
This description is a bit misleading for users.
Does this PR introduce any user-facing change?
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?