[SPARK-10683][SPARK-16510][SPARKR] Move SparkR include jar test to SparkSubmitSuite

[shivaram]

What changes were proposed in this pull request?

This change moves the include jar test from R to SparkSubmitSuite and uses a dynamically compiled jar. This helps us remove the binary jar from the R package and solves both the CRAN warnings and the lack of source being available for this jar.

How was this patch tested?

SparkR unit tests, SparkSubmitSuite, check-cran.sh

[shivaram]

cc @felixcheung

[SparkQA]

Test build #62441 has finished for PR 14243 at commit 64f95a4.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • |public class DummyClass implements java.io.Serializable

[SparkQA]

Test build #62443 has finished for PR 14243 at commit 4206104.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

cc @sun-rui

[felixcheung]

probably left behind from merge - sc was not referenced and was removed.

[shivaram]

If we remove sc then it prints a line

Java ref type org.apache.spark.sql.SparkSession id 1

It doesn't matter in a test case but I wonder if we should return something else / print something better for users who might call this.

[felixcheung]

we could change that in print.jobj? since we don't require specifying SparkSession in calls, we don't really need to return this from sparkR.session()?

[felixcheung]

Looks good, interesting approach - this won't run in CRAN since this is part of Scala tests?

[sun-rui]

Will this test be run always no matter if the "sparkr" profile is specified or not? In other words, does R need to installed for all spark tests to pass?

[shivaram]

@sun-rui Thats a good point. Right now the test will be canceled if R is not installed (from the line assume(R.isInstalled). I also added a check to make sure SparkR is installed in SPARK_HOME/R/lib. With this it cancels the test but passes the entire test suite if SparkR isn't installed. It prints a warning message which looks like

- include an external JAR in SparkR !!! CANCELED !!! (42 milliseconds)
[info]   org.apache.spark.api.r.RUtils.isSparkRInstalled was false SparkR is not installed in this build. (SparkSubmitSuite.scala:442)

@felixcheung Yeah thats the idea. CRAN will only run the R tests and not the Scala ones.

[felixcheung]

I guess it is harder to tell if the "R tests" are passing if it's embedded in Scala test, but that's an existing problem we have..

[SparkQA]

Test build #62474 has finished for PR 14243 at commit c131e9d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

better to add the SparkR installation check to the existing disable test?
ignore("correctly builds R packages included in a jar with --packages") { ...}

My expected solution would be setting an env var when the "sparkr" profile is specifed, and run SparkR related tests when the env var is detected. However, lets go with current way for now.

[shivaram]

Added the check to the ignore test case as well.

[SparkQA]

Test build #62542 has finished for PR 14243 at commit 47aea85.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

LGTM

[shivaram]

Thanks @sun-rui - Merging this to master and branch-2.0