[SPARKR][PYSPARK] Fix R source package name to match Spark version. Remove pip tar.gz from distribution

[shivaram]

What changes were proposed in this pull request?

Fixes name of R source package so that the cp in release-build.sh works correctly.

Issue discussed in #16014 (comment)

[shivaram]

cc @felixcheung @rxin - I tested this locally by running ./dev/make-distribution.sh --name "hadoop-2.6" --tgz --r -Phadoop-2.6 -Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos

[felixcheung]

LGTM. I guess this is an issue with snapshot builds.

[felixcheung]

I suspect though this might cause mismatch between the version in source package file name and the version in DESCRIPTION inside the package. Not sure if R CMD check might complain. R version cannot take the -SNAPSHOT part of the version string though.

[shivaram]

cc @holdenk @rxin - I also tried to fix the pip issue in this PR.

The main thing here is that we dont have PYSPARK_VERSION inside the make-distribution script so I just remove all pyspark*.tar.gz right now. Let me know if there is a better solution. I'm testing this locally now

[shivaram]

@holdenk this seems to work in my machine in that the ./python/dist/pyspark-2.1.1.dev0.tar.gz was removed from spark-2.1.1-SNAPSHOT-bin-hadoop-2.6.tgz that I built using the command[1].

One more question for you: Is it expected that the Spark dependency JARs are a part of the pip installable package ? i.e. when I look at the contents of say pyspark-2.1.0+hadoop2.7.tar.gz from [2], I find that it has all the Spark dependencies in pyspark-2.1.0+hadoop2.7/deps/jars/. I just wanted to check if that was the expected behavior

[1] ./dev/make-distribution.sh --name "hadoop-2.6" --tgz --pip --r -Phadoop-2.6 -Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos

[2]http://people.apache.org/~pwendell/spark-releases/spark-2.1.0-rc2-bin/pyspark-2.1.0+hadoop2.7.tar.gz

[holdenk]

Indeed that is the expected behavior, otherwise the user would have to install Spark regularly as well which would make it less useful.

…

On Fri, Dec 9, 2016 at 9:25 AM Shivaram Venkataraman < ***@***.***> wrote: @holdenk <https://github.com/holdenk> this seems to work in my machine in that the ./python/dist/pyspark-2.1.1.dev0.tar.gz was removed from spark-2.1.1-SNAPSHOT-bin-hadoop-2.6.tgz that I built using the command[1]. One more question for you: Is it expected that the Spark dependency JARs are a part of the pip installable package ? i.e. when I look at the contents of say pyspark-2.1.0+hadoop2.7.tar.gz from [2], I find that it has all the Spark dependencies in pyspark-2.1.0+hadoop2.7/deps/jars/. I just wanted to check if that was the expected behavior [1] ./dev/make-distribution.sh --name "hadoop-2.6" --tgz --pip --r -Phadoop-2.6 -Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16221 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADp9ex39RMyFDYXmJZ6uFS62tlB5tRsks5rGK37gaJpZM4LIZSj> .

[SparkQA]

Test build #69889 has finished for PR 16221 at commit 8a349f8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #69890 has finished for PR 16221 at commit c15916a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

Thanks @holdenk - I'm going to merge this as this script isn't tested by Jenkins. I will manually test this by triggering a nightly build in branch-2.1

[SparkQA]

Test build #69896 has finished for PR 16221 at commit 74d779b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #69897 has finished for PR 16221 at commit 9b97765.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

FYI the pip issue is fixed as you can see in the nightly build at http://people.apache.org/~pwendell/spark-nightly/spark-branch-2.1-bin/spark-2.1.1-SNAPSHOT-2016_12_08_18_31-ef5646b-bin/ --

spark-2.1.1-SNAPSHOT-bin-hadoop2.7.tgz	2016-12-09 02:52	187M	

Further the SparkR build was successful [1] but we are right now missing a line to copy the source archive with FTP - I am sending a PR for that

[1] https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.1-package/7/console

[felixcheung]

I tested a source package with a different version in filename vs DESCRIPTION and it seems to be working fine.