[SPARK-45777][CORE] Support spark.test.appId in LocalSchedulerBackend#43645
[SPARK-45777][CORE] Support spark.test.appId in LocalSchedulerBackend#43645dongjoon-hyun wants to merge 2 commits intoapache:masterfrom
spark.test.appId in LocalSchedulerBackend#43645Conversation
|
When you have some time, could you review this test conf PR, too, @LuciferYang ? |
| extends SchedulerBackend with ExecutorBackend with Logging { | ||
|
|
||
| private val appId = "local-" + System.currentTimeMillis | ||
| private val appId = sys.props.getOrElse("spark.test.appId", "local-" + System.currentTimeMillis) |
There was a problem hiding this comment.
Is there a particular reason to retrieve this from the system properties instead of the SparkConf?
There was a problem hiding this comment.
Although I did to avoid SparkConf(false) case, in my use case, I always use as environment variables and it will be okay to load from SparkConf. Do you want me to change, @yaooqinn ?
There was a problem hiding this comment.
From my perspective, it looks better to read from SparkConf if it's available in this case.
|
I addressed your comment, @yaooqinn . Could you review this once more? |
|
Also, thank you for review and approval, @LuciferYang . |
yaooqinn
left a comment
yaooqinn
left a comment
There was a problem hiding this comment.
LGTM, thanks for the updates
|
Thank you so much! I verified manually this test conf and updated the PR description. Merged to master! |
…end` ### What changes were proposed in this pull request? This PR aims to support `spark.test.appId` in `LocalSchedulerBackend` like the following. ``` $ bin/spark-shell --driver-java-options="-Dspark.test.appId=test-app-2023" ... Spark context available as 'sc' (master = local[*], app id = test-app-2023). ``` ``` $ bin/spark-shell -c spark.test.appId=test-app-2026 -c spark.eventLog.enabled=true -c spark.eventLog.dir=/Users/dongjoon/data/history ... Spark context available as 'sc' (master = local[*], app id = test-app-2026). ``` ### Why are the changes needed? Like the other `spark.test.*` configurations, this enables the developers control the appId in `LocalSchedulerBackend`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43645 from dongjoon-hyun/SPARK-45777. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
4 participants
What changes were proposed in this pull request?
This PR aims to support
spark.test.appIdinLocalSchedulerBackendlike the following.Why are the changes needed?
Like the other
spark.test.*configurations, this enables the developers control the appId inLocalSchedulerBackend.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual.
Was this patch authored or co-authored using generative AI tooling?
No.