Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-41958][CORE] Disallow arbitrary custom classpath with proxy user in cluster mode #39474

Closed
wants to merge 2 commits into from

Conversation

Ngone51
Copy link
Member

@Ngone51 Ngone51 commented Jan 10, 2023

What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

Does this PR introduce any user-facing change?

Yes. User should reenable this feature by spark.submit.proxyUser.allowCustomClasspathInClusterMode.

How was this patch tested?

Manually tested.

@github-actions github-actions bot added the CORE label Jan 10, 2023
@HyukjinKwon
Copy link
Member

Merged to master.

@mrmadira
Copy link

Hi - Is it possible to get a backporting to Spark 3.3 for this?

degant pushed a commit to degant/spark that referenced this pull request Jun 1, 2023
[SPARK-41958][CORE] Disallow arbitrary custom classpath with proxy user in cluster mode

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.toth@gmail.com>
Co-authored-by: Yi Wu <yi.wu@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

(cherry picked from commit 909da96)
dongjoon-hyun pushed a commit that referenced this pull request Jun 6, 2023
…xy user in cluster mode

Backporting fix for SPARK-41958 to 3.3 branch from #39474
Below description from original PR.

--------------------------

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes #39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.tothgmail.com>
Co-authored-by: Yi Wu <yi.wudatabricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223apache.org>

(cherry picked from commit 909da96)

### What changes were proposed in this pull request?

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes #41428 from degant/spark-41958-3.3.

Lead-authored-by: Degant Puri <depuri@microsoft.com>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
hellozepp pushed a commit to hellozepp/spark that referenced this pull request Oct 30, 2023
…xy user in cluster mode

Backporting fix for SPARK-41958 to 3.3 branch from apache#39474
Below description from original PR.

--------------------------

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.tothgmail.com>
Co-authored-by: Yi Wu <yi.wudatabricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223apache.org>

(cherry picked from commit 909da96)

### What changes were proposed in this pull request?

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes apache#41428 from degant/spark-41958-3.3.

Lead-authored-by: Degant Puri <depuri@microsoft.com>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
hellozepp added a commit to Kyligence/spark that referenced this pull request Oct 30, 2023
…xy user in cluster mode (#706)

Backporting fix for SPARK-41958 to 3.3 branch from apache#39474
Below description from original PR.

--------------------------

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.tothgmail.com>



(cherry picked from commit 909da96)

### What changes were proposed in this pull request?

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes apache#41428 from degant/spark-41958-3.3.

Lead-authored-by: Degant Puri <depuri@microsoft.com>

Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: Degant Puri <depuri@microsoft.com>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
yhcast0 pushed a commit to yhcast0/spark that referenced this pull request Nov 16, 2023
…xy user in cluster mode (Kyligence#706)

Backporting fix for SPARK-41958 to 3.3 branch from apache#39474
Below description from original PR.

--------------------------

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.tothgmail.com>



(cherry picked from commit 909da96)

### What changes were proposed in this pull request?

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes apache#41428 from degant/spark-41958-3.3.

Lead-authored-by: Degant Puri <depuri@microsoft.com>

Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: Degant Puri <depuri@microsoft.com>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
yhcast0 pushed a commit to Kyligence/spark that referenced this pull request Nov 16, 2023
…xy user in cluster mode (#706)

Backporting fix for SPARK-41958 to 3.3 branch from apache#39474
Below description from original PR.

--------------------------

### What changes were proposed in this pull request?

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

### Why are the changes needed?

To avoid arbitrary classpath in spark cluster.

### Does this PR introduce _any_ user-facing change?

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

### How was this patch tested?

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.tothgmail.com>



(cherry picked from commit 909da96)

### What changes were proposed in this pull request?

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes apache#41428 from degant/spark-41958-3.3.

Lead-authored-by: Degant Puri <depuri@microsoft.com>

Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: Degant Puri <depuri@microsoft.com>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
yhcast0 pushed a commit to yhcast0/spark that referenced this pull request Dec 19, 2023
…er in cluster mode

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

To avoid arbitrary classpath in spark cluster.

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.toth@gmail.com>
Co-authored-by: Yi Wu <yi.wu@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
yhcast0 added a commit to Kyligence/spark that referenced this pull request Dec 26, 2023
…er in cluster mode (#722)

This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default.

To avoid arbitrary classpath in spark cluster.

Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`.

Manually tested.

Closes apache#39474 from Ngone51/dev.

Lead-authored-by: Peter Toth <peter.toth@gmail.com>

Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Co-authored-by: Peter Toth <peter.toth@gmail.com>
Co-authored-by: Yi Wu <yi.wu@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
5 participants