Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-29574][K8S] Add SPARK_DIST_CLASSPATH to the executor class path #26493

Closed
wants to merge 3 commits into from
Closed

[SPARK-29574][K8S] Add SPARK_DIST_CLASSPATH to the executor class path #26493

wants to merge 3 commits into from

Conversation

sshakeri
Copy link

@sshakeri sshakeri commented Nov 13, 2019 •
edited by dongjoon-hyun

What changes were proposed in this pull request?

Include $SPARK_DIST_CLASSPATH in class path when launching CoarseGrainedExecutorBackend on Kubernetes executors using the provided entrypoint.sh

Why are the changes needed?

For user provided Hadoop, $SPARK_DIST_CLASSPATH contains the required jars.

Does this PR introduce any user-facing change?

no

How was this patch tested?

Kubernetes 1.14, Spark 2.4.4, Hadoop 3.2.1. Adding $SPARK_DIST_CLASSPATH to -cp param of entrypoint.sh enables launching the executors correctly.

@srowen
Copy link
Member

srowen commented Nov 13, 2019

This can't affect 2.4 if it affects Hadoop 3.x only, as there is no Hadoop 3 support in 2.4. Is this valid for master (only)? It needs a JIRA, etc. Please see https://spark.apache.org/contributing.html

@sshakeri sshakeri changed the title Fix Spark 2.4.4 on Kubernetes for Hadoop 3.2.1 [SPARK-29882][K8S] Fix Spark 2.4.4 on Kubernetes for Hadoop 3.2.1 Nov 13, 2019
@sshakeri
Copy link
Author

Thanks for the link I updated the pull request.

This issue exists for all the builds that use the -Phadoop-provided and -Pkubernetes profiles. Including the released 2.4.4 without-Hadoop version.

@sshakeri sshakeri changed the title [SPARK-29882][K8S] Fix Spark 2.4.4 on Kubernetes for Hadoop 3.2.1 [SPARK-29882][K8S] Fix Spark 2.4.4 on Kubernetes for without Hadoop builds Nov 13, 2019
@vanzin
Copy link
Contributor

vanzin commented Nov 13, 2019

There's already a bug filed about this: SPARK-29574

@srowen
Copy link
Member

srowen commented Nov 13, 2019

OK at least the title/description needs to be updated, but sounds like this could be a dupe of another JIRA and/or PR.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-29882][K8S] Fix Spark 2.4.4 on Kubernetes for without Hadoop builds [SPARK-29574][K8S] Fix Spark 2.4.4 on Kubernetes for without Hadoop builds Nov 13, 2019
@dongjoon-hyun
Copy link
Member

cc @holdenk

@dongjoon-hyun
Copy link
Member

ok to test

@SparkQA
Copy link

SparkQA commented Nov 13, 2019

Test build #113724 has finished for PR 26493 at commit d862d37.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 13, 2019

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/18603/

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-29574][K8S] Fix Spark 2.4.4 on Kubernetes for without Hadoop builds [SPARK-29574][K8S] Add SPARK_DIST_CLASSPATH to the executor class path Nov 13, 2019
@dongjoon-hyun
Copy link
Member

Hi, @sshakeri . Thank you for your first contribution. I updated the PR title to describe your proposal in this PR. (Also, with the previous JIRA ID)

@SparkQA
Copy link

SparkQA commented Nov 13, 2019

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/18603/

@vanzin
Copy link
Contributor

vanzin commented Nov 15, 2019

I don't see any code in the k8s backend that's actually propagating SPARK_DIST_CLASSPATH to the pods. That's generally propagated e.g. from spark-env.sh when you use YARN. Although I can see how that's a bit different on k8s (since the pod and launch node may be really different).

Anyway, it seems that at the very least hadoop-provided.md needs to be updated for this case, since the instructions there don't work for k8s as far as I can tell.

@sshakeri
Copy link
Author

sshakeri commented Nov 15, 2019 •
edited

The value of SPARK_DIST_CLASSPATH is derived from Hadoop during setup.

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

More info in the docs.

But interestingly the value that $(hadoop classpath) generates misses $HADOOP_HOME/share/hadoop/tools/lib/* I manually append that. You only need to do this if you depend on any of the drivers in tools like Azure (ABFS, ADLS etc.).

@vanzin
Copy link
Contributor

vanzin commented Nov 15, 2019

During what setup?

I'm familiar with the docs, thanks. The docs say to put that in your spark-env.sh. But k8s executors don't load spark-env-sh (the driver does because it's executed using spark-submit).

The docs also don't specify what spark-env.sh they're talking about. On YARN, that's my local file, from the submitting node. That will not work for k8s for the reasons I mentioned before.

@sshakeri
Copy link
Author

You would add that to your executor dockerfile.

@vanzin
Copy link
Contributor

vanzin commented Nov 15, 2019

Then, as I asked, could you add that to the docs?

@sshakeri
Copy link
Author

sshakeri commented Nov 15, 2019 •
edited

We have to update the dockerfile there too. I will submit an update.

@dongjoon-hyun
Copy link
Member

Hi, @sshakeri . Any updates?

@sshakeri
Copy link
Author

Hi @dongjoon-hyun I will post an update this weekend.

@dongjoon-hyun
Copy link
Member

Gentle ping, @sshakeri .

@sshakeri
Copy link
Author

sshakeri commented Dec 9, 2019

I added instructions on how to build the executor image that works with the Hadoop free version.

@SparkQA
Copy link

SparkQA commented Dec 9, 2019

Test build #115033 has finished for PR 26493 at commit 0035666.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 9, 2019

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19853/

@SparkQA
Copy link

SparkQA commented Dec 9, 2019

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19853/

vanzin
vanzin reviewed Dec 9, 2019
View reviewed changes
docs/hadoop-provided.md Outdated Show resolved Hide resolved
@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20083/

@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Test build #115273 has finished for PR 26493 at commit 8ab416f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20083/

@sshakeri
Copy link
Author

@dongjoon-hyun I think this is fine, can we rerun the checks? thanks

@vanzin
Copy link
Contributor

vanzin commented Dec 13, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Test build #115318 has finished for PR 26493 at commit 8ab416f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20126/

@SparkQA
Copy link

SparkQA commented Dec 13, 2019

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20126/

@vanzin
Copy link
Contributor

vanzin commented Dec 16, 2019

Only the flaky client deps test failed, so looks good. Merging to master.

@vanzin vanzin closed this in b573f23 Dec 16, 2019
@dongjoon-hyun
Copy link
Member

Thank you all!

This was referenced Oct 31, 2020
Closed
Closed
viirya pushed a commit that referenced this pull request Oct 31, 2020
@dungdm93 @viirya
[SPARK-29574][K8S][2.4] Add SPARK_DIST_CLASSPATH to the executor clas…
64a21ea 
…s path

### What changes were proposed in this pull request?

This is a backport of #26493 according to the community request #30174 .

Include `$SPARK_DIST_CLASSPATH` in class path when launching `CoarseGrainedExecutorBackend` on Kubernetes executors using the provided `entrypoint.sh`

### Why are the changes needed?

For user provided Hadoop, `$SPARK_DIST_CLASSPATH` contains the required jars.

### Does this PR introduce any user-facing change?
no

### How was this patch tested?
Kubernetes 1.14, Spark 2.4.4, Hadoop 3.2.1. Adding $SPARK_DIST_CLASSPATH to  `-cp ` param of entrypoint.sh enables launching the executors correctly.

Closes #30214 from dongjoon-hyun/SPARK-29574-2.4.

Lead-authored-by: Shahin Shakeri <shahin.shakeri@pwc.com>
Co-authored-by: Đặng Minh Dũng <dungdm93@live.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vanzin vanzin vanzin left review comments

Assignees
No one assigned
Labels
KUBERNETES
Projects
None yet
Milestone
No milestone
5 participants
@sshakeri @srowen @vanzin @dongjoon-hyun @SparkQA