[SPARK-22495] Fix setup of SPARK_HOME variable on Windows #19807
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit b58f740)
|
@HyukjinKwon I've created this cherry picked PR onto branch-2.2. Please take a look if this is what you had in mind. |
|
Test build #84142 has finished for PR 19807 at commit
|
|
Build started: [SparkR] |
|
LGTM pending AppVeyor tests. |
HyukjinKwon
reviewed
Nov 24, 2017
| ) | ||
|
|
||
| rem If there is python installed, trying to use the root dir as SPARK_HOME | ||
| where %PYTHON_RUNNER% > nul 2>$1 |
There was a problem hiding this comment.
There seems a typo here actually !! where %PYTHON_RUNNER% > nul 2>$1 -> where %PYTHON_RUNNER% > nul 2>&1.
Will fix it up in master by myself soon.
asfgit
pushed a commit
that referenced
this pull request
Nov 24, 2017
## What changes were proposed in this pull request? This is a cherry pick of the original PR 19370 onto branch-2.2 as suggested in #19370 (comment). Fixing the way how `SPARK_HOME` is resolved on Windows. While the previous version was working with the built release download, the set of directories changed slightly for the PySpark `pip` or `conda` install. This has been reflected in Linux files in `bin` but not for Windows `cmd` files. First fix improves the way how the `jars` directory is found, as this was stoping Windows version of `pip/conda` install from working; JARs were not found by on Session/Context setup. Second fix is adding `find-spark-home.cmd` script, which uses `find_spark_home.py` script, as the Linux version, to resolve `SPARK_HOME`. It is based on `find-spark-home` bash script, though, some operations are done in different order due to the `cmd` script language limitations. If environment variable is set, the Python script `find_spark_home.py` will not be run. The process can fail if Python is not installed, but it will mostly use this way if PySpark is installed via `pip/conda`, thus, there is some Python in the system. ## How was this patch tested? Tested on local installation. Author: Jakub Nowacki <j.s.nowacki@gmail.com> Closes #19807 from jsnowacki/fix_spark_cmds_2.
|
I fixed it during merge and merged this to 2.2. |
|
Thanks! Closing. |
MatthewRBruce
pushed a commit
to Shopify/spark
that referenced
this pull request
Jul 31, 2018
## What changes were proposed in this pull request? This is a cherry pick of the original PR 19370 onto branch-2.2 as suggested in apache#19370 (comment). Fixing the way how `SPARK_HOME` is resolved on Windows. While the previous version was working with the built release download, the set of directories changed slightly for the PySpark `pip` or `conda` install. This has been reflected in Linux files in `bin` but not for Windows `cmd` files. First fix improves the way how the `jars` directory is found, as this was stoping Windows version of `pip/conda` install from working; JARs were not found by on Session/Context setup. Second fix is adding `find-spark-home.cmd` script, which uses `find_spark_home.py` script, as the Linux version, to resolve `SPARK_HOME`. It is based on `find-spark-home` bash script, though, some operations are done in different order due to the `cmd` script language limitations. If environment variable is set, the Python script `find_spark_home.py` will not be run. The process can fail if Python is not installed, but it will mostly use this way if PySpark is installed via `pip/conda`, thus, there is some Python in the system. ## How was this patch tested? Tested on local installation. Author: Jakub Nowacki <j.s.nowacki@gmail.com> Closes apache#19807 from jsnowacki/fix_spark_cmds_2.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a cherry pick of the original PR 19370 onto branch-2.2 as suggested in #19370 (comment).
Fixing the way how
SPARK_HOMEis resolved on Windows. While the previous version was working with the built release download, the set of directories changed slightly for the PySparkpiporcondainstall. This has been reflected in Linux files inbinbut not for Windowscmdfiles.First fix improves the way how the
jarsdirectory is found, as this was stoping Windows version ofpip/condainstall from working; JARs were not found by on Session/Context setup.Second fix is adding
find-spark-home.cmdscript, which usesfind_spark_home.pyscript, as the Linux version, to resolveSPARK_HOME. It is based onfind-spark-homebash script, though, some operations are done in different order due to thecmdscript language limitations. If environment variable is set, the Python scriptfind_spark_home.pywill not be run. The process can fail if Python is not installed, but it will mostly use this way if PySpark is installed viapip/conda, thus, there is some Python in the system.How was this patch tested?
Tested on local installation.