Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-13278][CORE] Launcher fails to start with JDK 9 EA #11160

Closed
wants to merge 5 commits into from

Conversation

cl4es
Copy link
Contributor

@cl4es cl4es commented Feb 10, 2016

See http://openjdk.java.net/jeps/223 for more information about the JDK 9 version string scheme.

@zsxwing
Copy link
Member

zsxwing commented Feb 10, 2016

ok to test

@zsxwing
Copy link
Member

zsxwing commented Feb 10, 2016

LGTM pending tests.

* Get the major version of the java.version string supplied.
*/
static int javaMajorVersion(String javaVersion) {
String[] version = javaVersion.split("[+.\\-]+");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a super minor point, but I don't think we need to split on anything other than "." even with non-JEPS223 strings since we only want to extract the major version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for early access version, e.g., 9-ea, the previous codes will throw an exception.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But they won't since we only try and access the second element of the array if the first element is less than or equal to 1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 9-ea the original split would give you an array whose first element is "9-ea", Integer.parseInt("9-ea") throws NumberFormatException. Feel free to try revert to the old regex and run the test cases I added to verify this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant Integer.parseInt("9-ea") will throw NumberFormatException.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yah sorry not enough coffee for me today, I guess we need .- but not + (although if we did use java.runtime.version we would need the +) and one of the samples in the test "9+100" shouldn't show up in java.version but should in java.runtime.version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, you're right that handling + is probably superfluous for java.version. I can prune that from the commit if you want, but doesn't seem to do any harm to leave it in for completeness.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably isn't much harm; the only concern would be that it split string, the test and the comment together imply a different input format than the one specified. It might be slightly nicer to have it match the spec since its just a few characters difference. (but a super minor point)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about just splitting on non-numbers? It's all kind of a theoretical difference though. This looks OK.

@SparkQA
Copy link

SparkQA commented Feb 11, 2016

Test build #51068 has finished for PR 11160 at commit 0da18a6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Feb 11, 2016

I think SparkBuild.scala has a similar computation that needs a similar treatment. Also test("Kill process") in UtilsSuite.scala.

@zsxwing
Copy link
Member

zsxwing commented Feb 12, 2016

@cl4es could you also fix the places mentioned by @srowen ?

@cl4es
Copy link
Contributor Author

cl4es commented Feb 12, 2016

I tried building with JDK 9 EA, and there are more immediate issues such as -XX:MaxPermSize (which was deprecated/removed in 8 and fails to start in 9) being used in various places. The quick fix would be to add -XX:+IgnoreUnrecognizedVMOptions, but that might be unsanitary. I can give fixing the version string logic a shot, though.

Runtime support is of a more immediate concern to us, though, as it allows the OpenJDK project to use Spark itself as a functional test/benchmark. Sadly JEP-223 has stopped that effort for now.

@cl4es
Copy link
Contributor Author

cl4es commented Feb 13, 2016

While not a general rule, testing lexically that version >= "1.8.0" like in UtilsSuite is actually OK for all current and all possible JEP-223 versions. I fixed SparkBuild.scala to use this trick, and gave the comment to javaMajorVersion an overhaul.

@SparkQA
Copy link

SparkQA commented Feb 13, 2016

Test build #51221 has finished for PR 11160 at commit 68159d8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

val Array(major, minor, _) = System.getProperty("java.version").split("\\.", 3)
if (major.toInt >= 1 && minor.toInt >= 8) Seq("-Xdoclint:all", "-Xdoclint:-missing") else Seq.empty
val version = System.getProperty("java.version")
if (version >= "1.8.0") Seq("-Xdoclint:all", "-Xdoclint:-missing") else Seq.empty
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this only happens to work for 1.8 vs 9. This seems hacky compared to just emulating the simple logic you wrote above -- why not that?

@srowen
Copy link
Member

srowen commented Feb 13, 2016

I think UtilsSuite still needs a treatment.

I think the code and scripts attempt to not set MaxPermSize for Java 8 already. If it doesn't somewhere, that also needs to be patched up. But my impression is this should already be the case

@cl4es
Copy link
Contributor Author

cl4es commented Feb 13, 2016

This should likely be refactored to a place where both sbt and test code can access it, but I hope this is good enough.

@SparkQA
Copy link

SparkQA commented Feb 13, 2016

Test build #51241 has finished for PR 11160 at commit 8220484.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Feb 14, 2016

LGTM

@srowen
Copy link
Member

srowen commented Feb 14, 2016

Merged to master

@asfgit asfgit closed this in 22e9723 Feb 14, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants