[SPARK-33408][K8S][R][3.0] Use R 3.6.3 in K8s R image

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade K8s R image to use R 3.6.3 which is the same version installed in Jenkins Servers.

Why are the changes needed?

Jenkins Server is using R 3.6.3.

+ SPARK_HOME=/home/jenkins/workspace/SparkPullRequestBuilder-K8s
+ /usr/bin/R CMD check --as-cran --no-tests SparkR_3.1.0.tar.gz
* using log directory ‘/home/jenkins/workspace/SparkPullRequestBuilder-K8s/R/SparkR.Rcheck’
* using R version 3.6.3 (2020-02-29)

OpenJDK docker image is using R 3.5.2 (2018-12-20) which is old and currently spark-3.0.1 fails to run SparkR.

$ cd spark-3.0.1-bin-hadoop3.2

$ bin/docker-image-tool.sh -R kubernetes/dockerfiles/spark/bindings/R/Dockerfile -n build

$ bin/spark-submit --master k8s://https://192.168.64.49:8443 --deploy-mode cluster --conf spark.kubernetes.container.image=spark-r:latest local:///opt/spark/examples/src/main/r/dataframe.R

$ k logs dataframe-r-b1c14b75b0c09eeb-driver
...
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.17.0.4 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.RRunner local:///opt/spark/examples/src/main/r/dataframe.R
20/11/10 06:03:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error: package or namespace load failed for ‘SparkR’ in rbind(info, getNamespaceInfo(env, "S3methods")):
 number of columns of matrices must match (see arg 2)
In addition: Warning message:
package ‘SparkR’ was built under R version 4.0.2
Execution halted

Does this PR introduce any user-facing change?

How was this patch tested?

Pass K8s IT.

[HyukjinKwon]

Looks good once the tests pass. I guess the fix will also go to the master, right?

Also, I think we can just drop 3.5 and change the minimum R version to 3.6 as well if this doesn't work. Should change the files fixed in #28908. If I remember correctly, I manually tested it only with R 3.6. R 3.5 was not tested - I just assumed that it works based on the documentation.

[dongjoon-hyun]

Thank you, @HyukjinKwon . I made a patch for master before this. Here~

• [SPARK-33408][SPARK-32354][K8S][R] Use R 3.6.3 in K8s R image and re-enable RTestsSuite #30130

Yes, although I tested this manually, we need to pass K8s CI.

[dongjoon-hyun]

This PR is using R 4.0 in K8s R image only. Technically, this depends on the base-OS image used openjdk docker image. So, I believe it will be better to install R 4.0 explicitly.

[SparkQA]

Test build #130963 has finished for PR 30310 at commit 3babd50.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35569/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35569/

[dongjoon-hyun]

Hi, @HyukjinKwon .
Could you review this again?

[dongjoon-hyun]

Hi, @viirya
Could you review this?

[HyukjinKwon]

Merged to branch-3.0.

[viirya]

lgtm

[dongjoon-hyun]

Thank you, @HyukjinKwon and @viirya !