[SPARK-3061] Fix Maven build under Windows

[JoshRosen]

The Maven build was failing on Windows because it tried to call the unix unzip utility to extract the Py4J files into core's build directory. I've fixed this issue by using the maven-antrun-plugin to perform the unzipping.

I also fixed an issue that prevented tests from running under Windows:

In the Maven ScalaTest plugin, the filename listed in is placed under the ; the current code places it in a subdirectory of reportsDirectory, e.g.

${project.build.directory}/surefire-reports/${project.build.directory}/SparkTestSuite.txt

This caused problems under Windows because it would try to create a subdirectory named "c:".

Note that the tests still fail under Windows (for other reasons); this PR just allows them to run and fail rather than crash when trying to create the test reports directory.

[SparkQA]

QA tests have started for PR 2165 at commit e347668.

• This patch merges cleanly.

[andrewor14]

nit: 2 spaces instead of 4

[SparkQA]

QA tests have started for PR 2165 at commit fbf3e61.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2165 at commit e347668.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)

[SparkQA]

QA tests have finished for PR 2165 at commit fbf3e61.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

test this please

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2165 at commit 651d210.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2165 at commit 651d210.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

I tested this on Windows and it works there. I still have to test PySpark on Yarn with this PR because it has the potential to break things there, though from a cursory glance the changes seem fine.

[JoshRosen]

@andrewor14 Unless the new unzipping code is somehow corrupting the files or unzipping them to a different location, won't it be sufficient to check that the same files are present in the assembly JAR without having to run an actual YARN cluster?

[andrewor14]

I suppose so. I am extra cautious about this because the whole PySpark on YARN thing is somewhat fickle. On a related note I was able to verify that the python files were actually there.

[srowen]

Re: can this break something -- we have seen in recent memory how zip behavior can be inconsistent with archives of >= 65536 files, which comes up with Spark assembly jars. I assume that's not true of the .zip file being unzipped here though. In that case this does seem like something that will work everywhere if it works anywhere.

[andrewor14]

LGTM. I tested this extensively locally on Windows and OSX, and remotely on a HDP cluster and on a standalone cluster. PySpark works as expected on all environments. I'll merge this once I discuss with @pwendell on what should go into branch-1.1 at this point.

[pwendell]

Hey @andrewor14 you can merge this into master and just leave the JIRA as unresolved and add targetVersion 1.1.1 and fixVersion 1.2.0 and make it a blocker. That way it will be clear it needs to be backported.

[andrewor14]

Ok, I will add a reminder for myself to back port this after the release. Thanks Josh.