Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-13278][CORE] Launcher fails to start with JDK 9 EA #11160

Closed
wants to merge 5 commits into from
Closed

[SPARK-13278][CORE] Launcher fails to start with JDK 9 EA #11160

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0da18a6
[SPARK-13278][CORE] Launcher fails to start with JDK 9 EA
cl4es Feb 10, 2016
c3ef836
Improve javadoc of javaMajorVersion
cl4es Feb 12, 2016
33b2376
Make SparkBuild.scala JEP-223 compliant
cl4es Feb 12, 2016
68159d8
Simplify fix since lexical comparison is sufficient to assert this
cl4es Feb 13, 2016
8220484
Better scala implementation for determining major java version
cl4es Feb 13, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,11 +322,9 @@ static void addPermGenSizeOpt(List<String> cmd) {
if (getJavaVendor() == JavaVendor.IBM) {
return;
}
String[] version = System.getProperty("java.version").split("\\.");
if (Integer.parseInt(version[0]) > 1 || Integer.parseInt(version[1]) > 7) {
if (javaMajorVersion(System.getProperty("java.version")) > 7) {
return;
}

for (String arg : cmd) {
if (arg.startsWith("-XX:MaxPermSize=")) {
return;
Expand All @@ -336,4 +334,18 @@ static void addPermGenSizeOpt(List<String> cmd) {
cmd.add("-XX:MaxPermSize=256m");
}

/**
* Get the major version of the java.version string supplied.
*/
static int javaMajorVersion(String javaVersion) {
String[] version = javaVersion.split("[+.\\-]+");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a super minor point, but I don't think we need to split on anything other than "." even with non-JEPS223 strings since we only want to extract the major version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for early access version, e.g., 9-ea, the previous codes will throw an exception.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But they won't since we only try and access the second element of the array if the first element is less than or equal to 1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 9-ea the original split would give you an array whose first element is "9-ea", Integer.parseInt("9-ea") throws NumberFormatException. Feel free to try revert to the old regex and run the test cases I added to verify this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant Integer.parseInt("9-ea") will throw NumberFormatException.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yah sorry not enough coffee for me today, I guess we need .- but not + (although if we did use java.runtime.version we would need the +) and one of the samples in the test "9+100" shouldn't show up in java.version but should in java.runtime.version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, you're right that handling + is probably superfluous for java.version. I can prune that from the commit if you want, but doesn't seem to do any harm to leave it in for completeness.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably isn't much harm; the only concern would be that it split string, the test and the comment together imply a different input format than the one specified. It might be slightly nicer to have it match the spec since its just a few characters difference. (but a super minor point)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about just splitting on non-numbers? It's all kind of a theoretical difference though. This looks OK.

int major = Integer.parseInt(version[0]);
// if major > 1, we're using the JEP-223 version string, e.g., 9-ea, 9+120
// otherwise the second number is the major version
if (major > 1) {
return major;
} else {
return Integer.parseInt(version[1]);
}
}
}
12 changes: 12 additions & 0 deletions launcher/src/test/java/org/apache/spark/launcher/CommandBuilderUtilsSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,18 @@ public void testPythonArgQuoting() {
assertEquals("\"a \\\"b\\\" c\"", quoteForCommandString("a \"b\" c"));
}

@Test
public void testJavaMajorVersion() {
assertEquals(6, javaMajorVersion("1.6.0_50"));
assertEquals(7, javaMajorVersion("1.7.0_79"));
assertEquals(8, javaMajorVersion("1.8.0_66"));
assertEquals(9, javaMajorVersion("9-ea"));
assertEquals(9, javaMajorVersion("9+100"));
assertEquals(9, javaMajorVersion("9"));
assertEquals(9, javaMajorVersion("9.1.0"));
assertEquals(10, javaMajorVersion("10"));
}

private void testOpt(String opts, List<String> expected) {
assertEquals(String.format("test string failed to parse: [[ %s ]]", opts),
expected, parseOptionString(opts));
Expand Down