Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spark_connect() should handle openjdk 11 gracefully #1922

Closed
sz-cgt opened this issue Feb 20, 2019 · 1 comment · Fixed by #2253
Closed

spark_connect() should handle openjdk 11 gracefully #1922

sz-cgt opened this issue Feb 20, 2019 · 1 comment · Fixed by #2253

Comments

@sz-cgt
Copy link

sz-cgt commented Feb 20, 2019

Error:

sc <- sparklyr::spark_connect(master = "local[*]")
#> Error in validate_java_version_line(master, version): Java version detected but couldn't parse version from: openjdk version "11.0.2" 2019-01-15

Created on 2019-02-20 by the reprex package (v0.2.1)

Root Cause:
The version string for openjdk 10 and above includes a release date. Starting with openjdk 11.0.2, that release date is in 2019. validate_java_version_line() fails to handle dates in 2019 (and presumably earlier release dats with a 9 in the month or the day) due to the following code

splatVersion <- splat[grepl("9|[0-9]+\\.[0-9]+\\.[0-9]+", splat)]

if the intention of that regexp is to match jdk 9 versions strings, it should either have a caret (^) before the 9 or something after it to prevent matches against a ISO date string.

Given that java 9 is end of life, a better answer is simply to remove the 9| from the regexp entirely

@zero323
Copy link
Contributor

zero323 commented Mar 11, 2019

@sz-cgt At the moment (Spark 2.4) running Spark on JDK9 or later is not supported.

AFAIK SPARK-24417 Build and Run Spark on JDK11 - serves as an umbrella ticket for different JDK compatibility issues, and informally supporting new JDKs should happen in 3.0 (time for Apache Spark 3.0?, and target versions for remaining sub-tasks).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants