build: Improve stability of macOS Spark 4 tests

[andygrove]

Which issue does this PR close?

Closes #1949

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.59%. Comparing base (f09f8af) to head (e2f809d).
⚠️ Report is 793 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2959      +/-   ##
============================================
+ Coverage     56.12%   59.59%   +3.47%     
- Complexity      976     1381     +405     
============================================
  Files           119      167      +48     
  Lines         11743    15492    +3749     
  Branches       2251     2568     +317     
============================================
+ Hits           6591     9233    +2642     
- Misses         4012     4959     +947     
- Partials       1140     1300     +160     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

macOS build still failing:

- Parquet temporal types written as INT96 (native_datafusion, native shuffle) (1 minute, 6 seconds)
- Parquet temporal types written as INT96 (native_iceberg_compat, native shuffle) (45 seconds, 530 milliseconds)
Warning: [395.069s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 4096k, guardsize: 16k, detached.
Warning: [395.069s][warning][os,thread] Failed to start the native thread for java.lang.Thread "shuffle-exchange-3604"
*** RUN ABORTED *** (6 minutes, 29 seconds)
  java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
  at java.base/java.lang.Thread.start0(Native Method)
  at java.base/java.lang.Thread.start(Thread.java:809)
  at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
  at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353)
  at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
  at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782)
  at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005)
  at org.apache.spark.sql.execution.SQLExecution$.withThreadLocalCaptured(SQLExecution.scala:333)
  at org.apache.spark.sql.execution.exchange.ShuffleExchangeLike.org$apache$spark$sql$execution$exchange$ShuffleExchangeLike$$triggerFuture(ShuffleExchangeExec.scala:87)
  at org.apache.spark.sql.execution.exchange.ShuffleExchangeLike.org$apache$spark$sql$execution$exchange$ShuffleExchangeLike$$triggerFuture$(ShuffleExchangeExec.scala:85)

[andygrove]

Possibly related: https://issues.apache.org/jira/browse/SPARK-51365

[manuzhang]

Possibly related: https://issues.apache.org/jira/browse/SPARK-51365

@andygrove I see SPARK-51365 has already been fixed in Spark 4.0.0