[SPARK-37663][CORE][TESTS] Mitigate ConcurrentModificationException thrown from tests in SparkContextSuite

[sarutak]

What changes were proposed in this pull request?

This PR fixes an issue that some tests in SparkContextSuite can throw ConcurrentModificationException with Scala 2.13.
https://github.com/apache/spark/runs/4543047740?check_suite_focus=true#step:9:20851
The cause seems to be same as SPARK-37315 (#34583).

Scala 2.13.7 includes an update to detect ConcurrentModificationException more precisely.
scala/scala#9786

You can easily reproduce this issue by applying the following diff to master.

diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index bc809f11cc..e5dde84c6e 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -1130,9 +1130,11 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu
       sc.addJar("ivy://org.apache.hive:hive-storage-api:2.7.0?" +
         "invalidParam1=foo&invalidParam2=boo")
       assert(sc.listJars().exists(_.contains("org.apache.hive_hive-storage-api-2.7.0.jar")))
-      assert(logAppender.loggingEvents.exists(_.getRenderedMessage.contains(
-        "Invalid parameters `invalidParam1,invalidParam2` found in Ivy URI query " +
-          "`invalidParam1=foo&invalidParam2=boo`.")))
+      assert(logAppender.loggingEvents.exists { x =>
+        Thread.sleep(1000)
+        x.getRenderedMessage.contains(
+          "Invalid parameters `invalidParam1,invalidParam2` found in Ivy URI query " +
+            "`invalidParam1=foo&invalidParam2=boo`.")})
     }
   }

Why are the changes needed?

Fix the flaky test.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Locally checked that no ConcurrentModificationException is thrown even if a sleep is inserted like the diff shown above.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50756/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50756/

[SparkQA]

Test build #146282 has finished for PR 34922 at commit 46dcd08.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50780/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50780/

[SparkQA]

Test build #146308 has finished for PR 34922 at commit 936fc34.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• (defaultdict(<class 'list'>,
• class PandasSQLStringFormatter(string.Formatter):
• class SQLStringFormatter(string.Formatter):
• class ExecutorRollPlugin extends SparkPlugin
• class ExecutorRollDriverPlugin extends DriverPlugin with Logging
• case class MapContainsKey(
• case class TryElementAt(left: Expression, right: Expression, child: Expression)
• case class ConvertTimezone(
• case class AesEncrypt(
• case class AesDecrypt(

[dongjoon-hyun]

+1, LGTM.