[SPARK-57952][CORE][TESTS] Retry retryOnOOM a few times to reduce SorterSuite flaky OOM

[HyukjinKwon]

What changes were proposed in this pull request?

Make the test helper retryOnOOM (in SparkFunSuite) retry a few times (default 3) with
a full runGC() between attempts, instead of running GC and retrying only once. The sole
caller (SorterSuite) is updated to the new retryOnOOM()(body) form.

Why are the changes needed?

The core module test job intermittently aborts with an OOM (observed ~1 in 5 runs, e.g.
in the branch-4.x Maven build):

*** RUN ABORTED ***
  An exception or error caused a run to abort: Java heap space
  java.lang.OutOfMemoryError: Java heap space
  at org.apache.spark.util.collection.SorterSuite.$anonfun$new$12(SorterSuite.scala:145)
  at org.apache.spark.SparkFunSuite.retryOnOOM(SparkFunSuite.scala:122)

SorterSuite allocates a single ~1 GB byte[] (heap is -Xmx4g). SPARK-57037 already
wrapped that allocation in retryOnOOM, which does one System.gc() + one retry. But a
single GC after the first failure does not always reclaim the previous test's large array
in time, so the lone retry allocates against a still-pressured heap and OOMs again.

Because the failure is intermittent (the job passes ~4 of 5 runs), 4 GB is not fundamentally
too small — this is transient heap pressure, exactly the case where additional GC+retry
attempts help. Retrying up to 3 times (with runGC(), which waits for a weak reference to
clear, between attempts) gives the JVM more chances to reclaim memory.

This is a test flakiness reduction, not a correctness change: a genuinely too-large or
leaked allocation still fails once the attempts are exhausted, so real OOMs are not masked.

Does this PR introduce any user-facing change?

No. Test-only change.

How was this patch tested?

SorterSuite (which exercises retryOnOOM) on a fork via GitHub Actions; confirmed the
signature change compiles and the suite passes.

CI evidence

• Failing run (apache/spark, branch-4.x build_maven, core module — SorterSuite OOM):
  https://github.com/apache/spark/actions/runs/28747189185
• Passing validation (fork, compiles core + runs SorterSuite):
  https://github.com/HyukjinKwon/spark-agent6/actions/runs/28766544512

Note: the underlying OOM is intermittent (~1/5), so this reduces the flake probability;
it is not claimed to statistically eliminate it.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

[HyukjinKwon]

Merge Summary:

• merged into master 5ae5aa5
• merged into branch-4.x 05158fc
• merged into branch-4.2 7322c77

Posted by merge_spark_pr.py