Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows #19310

Closed
wants to merge 1 commit into from
Closed

[SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows #19310

wants to merge 1 commit into from

Conversation

jsnowacki
Copy link
Contributor

What changes were proposed in this pull request?

Fix for setup of SPARK_JARS_DIR on Windows as it looks for %SPARK_HOME%\RELEASE file instead of %SPARK_HOME%\jars as it should. RELEASE file is not included in the pip build of PySpark.

How was this patch tested?

Local install of PySpark on Anaconda 4.4.0 (Python 3.6.1).

@felixcheung
Copy link
Member

jenkins, test this please

felixcheung
felixcheung approved these changes Sep 23, 2017
View reviewed changes
Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this definitely seems like a problem
LGTM, but I'll add more folks to review

@felixcheung
Copy link
Member

felixcheung commented Sep 23, 2017 •
edited
Loading

@holdenk @ueshin @HyukjinKwon as it says python pip package, but I'm pretty sure this is cross language

@felixcheung
Copy link
Member

I'm a little puzzled how this wouldn't broken on windows without this fix with release build... need to run some tests to see.

@SparkQA
Copy link

SparkQA commented Sep 23, 2017

Test build #82110 has finished for PR 19310 at commit b5f8ae6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

HyukjinKwon
HyukjinKwon approved these changes Sep 23, 2017
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM too. I tested

@HyukjinKwon
Copy link
Member

@jsnowacki, I am merging this assuming you will fix up https://issues.apache.org/jira/browse/SPARK-18136?focusedCommentId=16175276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16175276 soon in a followup. I also verified this too.

In a quick look, we probably should fix codes around

spark/python/setup.py

Line 126 in aad2125

copytree(SCRIPTS_PATH, SCRIPTS_TARGET)
I guess

asfgit pushed a commit that referenced this pull request Sep 23, 2017
@jsnowacki @HyukjinKwon
[SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows
c0a34a9 
## What changes were proposed in this pull request?

Fix for setup of `SPARK_JARS_DIR` on Windows as it looks for `%SPARK_HOME%\RELEASE` file instead of `%SPARK_HOME%\jars` as it should. RELEASE file is not included in the `pip` build of PySpark.

## How was this patch tested?

Local install of PySpark on Anaconda 4.4.0 (Python 3.6.1).

Author: Jakub Nowacki <j.s.nowacki@gmail.com>

Closes #19310 from jsnowacki/master.

(cherry picked from commit c11f24a)
Signed-off-by: hyukjinkwon <gurwls223@gmail.com>
asfgit pushed a commit that referenced this pull request Sep 23, 2017
@jsnowacki @HyukjinKwon
[SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows
0b3e7cc 
## What changes were proposed in this pull request?

Fix for setup of `SPARK_JARS_DIR` on Windows as it looks for `%SPARK_HOME%\RELEASE` file instead of `%SPARK_HOME%\jars` as it should. RELEASE file is not included in the `pip` build of PySpark.

## How was this patch tested?

Local install of PySpark on Anaconda 4.4.0 (Python 3.6.1).

Author: Jakub Nowacki <j.s.nowacki@gmail.com>

Closes #19310 from jsnowacki/master.

(cherry picked from commit c11f24a)
Signed-off-by: hyukjinkwon <gurwls223@gmail.com>
@HyukjinKwon
Copy link
Member

Merged to master, branch-2.2 and branch-2.1.

@asfgit asfgit closed this in c11f24a Sep 23, 2017
@holdenk
Copy link
Contributor

holdenk commented Sep 23, 2017

It probably is. The feature only works on Linux, but thanks for working towards fixing this.

MatthewRBruce pushed a commit to Shopify/spark that referenced this pull request Jul 31, 2018
@jsnowacki @MatthewRBruce
[SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows
e5576a0 
## What changes were proposed in this pull request?

Fix for setup of `SPARK_JARS_DIR` on Windows as it looks for `%SPARK_HOME%\RELEASE` file instead of `%SPARK_HOME%\jars` as it should. RELEASE file is not included in the `pip` build of PySpark.

## How was this patch tested?

Local install of PySpark on Anaconda 4.4.0 (Python 3.6.1).

Author: Jakub Nowacki <j.s.nowacki@gmail.com>

Closes apache#19310 from jsnowacki/master.

(cherry picked from commit c11f24a)
Signed-off-by: hyukjinkwon <gurwls223@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@felixcheung felixcheung felixcheung approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jsnowacki @felixcheung @SparkQA @HyukjinKwon @holdenk