New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap #31428
Conversation
Could you review this, @ScrapCodes ? There was a user request to disable the new feature of SPARK-30985 for some security reasons. |
Test build #134750 has finished for PR 31428 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
Could you review this, @holdenk ? |
Hi @dongjoon-hyun , Thank you for the PR. You mentioned due to a security issue, you would like add this config. Can you please describe the issue either here or if you think it makes sense then @Private list. I have taken a look a cursory look, no issues found. :) |
Ok, +1 LGTM ! @dongjoon-hyun, Feel free to merge this. |
Thank you, @ScrapCodes . Yes, for security, some namespaces do not allow the driver pod to create new ConfigMaps. As a result, Spark jobs are unable to be executed in those namespaces. This unblocks Spark jobs in those namespace. As I wrote in the PR description, this doesn't aim to hurt the functionality of the original contribution. Instead, this PR aims to provide a workaround for those security enforced namespace. |
Merged to master. |
Ahh, makes sense then. Wish, we could autodetect and fallback too. |
### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closes apache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closes apache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit f66e38c) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Hi, @dongjoon-hyun and @ScrapCodes, When I was developing #41196, I initially thought it may be desirable to reuse the However after some digging and found some limitation exposed by K8S:
Therefore I think there should be some rules when developing feature steps for spark on K8S regarding config maps:
I'd like to make a proposal here:
WDYT? |
Yes, actually, that bit us as you see in this PR's goal.
Please send an email to dev@spark mailing list for further Q&A or discussion. We don't want to open a long discussion on the existing PR like this because that frequently causes unnecessary confusion for other audiences, @advancedxy . |
Thanks. Let me open a discuss mail on the dev mail list. |
Thank you! |
What changes were proposed in this pull request?
This PR aims to add a new configuration
spark.kubernetes.executor.disableConfigMap
.Why are the changes needed?
This can be use to disable config map creating for executor pods due to #27735 .
Does this PR introduce any user-facing change?
No. By default, this doesn't change AS-IS behavior.
This is a new feature to add an ability to disable SPARK-30985.
How was this patch tested?
Pass the newly added UT.