[SPARK-37293][TESTS] Remove explicit GC options from Scala tests

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to remove the explicit GC options in Scala tests.

Why are the changes needed?

At Apache Spark 3.0, SPARK-29282 introduced the explicit GC options in Scala tests to see the consistent result between Java8/Java11. Now, we have Java 17 additionally and SPARK-37120 added Java 11/17 GitHub Action jobs.

This PR aims to use the default GC options for each JVM. This is a partial revert of SPARK-29282.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the GitHub Action CIs without memory issues.

[dongjoon-hyun]

Hi, @HyukjinKwon and @LuciferYang .
The daily Java11 and 17 runs seems to work, but the only exception looks like the known the memory issue.
So I hope this try can mitigate the situation at least the following warning.

[warn] In the last 3393 seconds, 6.757 (0.2%) were spent in GC. [Heap: 3.49GB free of 4.00GB, max 4.00GB]
Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.

• https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49586/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49586/

[SparkQA]

Test build #145117 has finished for PR 34560 at commit a3103a2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you review this, @mridulm ?

[HyukjinKwon]

My only concern is the tests being flaky - the current GA builds are flaky because of hitting memory issues. I vaguely remember that using UseParallelGC arguably less reaches the memory limit (I did take a quick look before).

I am going to approve this but we might have to monitor the tests for the time being..

[LuciferYang]

LGTM +1

[dongjoon-hyun]

Thanks. Yes, GA jobs are flaky due to the memory limits and I'll monitor this carefully.

Note that this will affect only Java 11/Java 17 which means

• Java11 build job (without tests)
• Java17 build job (without tests)
• Daily Java11/17 build and test jobs which is added recently.

The other jobs are running Java 8 whose default GC is ParallelGC. In addition, the flakiness mainly occurs at the build&test jobs.

Especially, this PR aims to stablize the daily Java11 and Java17 jobs. I'll keep improving those test coverage for Spark 3.3..

[dongjoon-hyun]

Thank you, @LuciferYang , too. Merged to master.