[SPARK-29603][YARN] Support application priority for YARN priority scheduling #26255
Conversation
|
Test build #112662 has finished for PR 26255 at commit
|
|
Test build #112665 has finished for PR 26255 at commit
|
|
cc @jerryshao @vanzin @cloud-fan, thanks in advance |
|
@yaooqinn Might point out in the config documentation that FIFO policy is required for priority (it is default in CapacityScheduler, but could be set to FAIR). |
yaooqinn
changed the title
|
Test build #112747 has finished for PR 26255 at commit
|
docs/running-on-yarn.md
Outdated
| @@ -419,6 +419,15 @@ To use a custom metrics.properties for the application master and executors, upd | |||
| in YARN ApplicationReports, which can be used for filtering when querying YARN apps. | |||
| </td> | |||
| </tr> | |||
| <tr> | |||
| <td><code>spark.yarn.app.priority</code></td> | |||
There was a problem hiding this comment.
We don't have any configs that use .app, so I'd call this just spark.yarn.priority.
docs/running-on-yarn.md
Outdated
| <td>(none)</td> | ||
| <td> | ||
| Application priority for YARN to define pending applications ordering policy, those with higher | ||
| integer value have a better opportunity to be activated. Currently Application priority is supported |
There was a problem hiding this comment.
"Currently, YARN only supports application priority when using FIFO ordering policy."
|
|
||
| sparkConf.get(APPLICATION_PRIORITY) match { | ||
| case Some(v) => appContext.setPriority(Priority.newInstance(v)) | ||
| case None => logDebug(s"${APPLICATION_PRIORITY.key} is not set. " + |
There was a problem hiding this comment.
This debug message is not really useful. Just do sparkConf.get(...).foreach.
|
Test build #113300 has finished for PR 26255 at commit
|
|
Merging to master. |
What changes were proposed in this pull request?
Priority for YARN to define pending applications ordering policy, those with higher priority have a better opportunity to be activated. YARN CapacityScheduler only.
Why are the changes needed?
Ordering pending spark apps
Does this PR introduce any user-facing change?
add a conf
How was this patch tested?
add ut