spark-ci walltime improvements

[kevinjqliu]

Context

spark-ci dominates the overall time CI takes to run. Based on most recent stats, it averages 1h5m while the next slowest ci job (flink-ci) averages around only 21m.
This means PRs must wait at least 1h for spark-ci to complete before merging.

A couple of things we can do to improve the overall time it takes to run spark-ci

1. fix the gradle cache. reduce time to download dependencies. fixed in Build: Designate a single Gradle cache writer across CI workflows #16356
2. parallelize execution. enabling -DtestParallelism=auto for all tests, following flink-ci. And split spark-ci into 2 concurrent jobs. Fixed in Build: Speed up Spark CI with parallel test execution #16357. This reduced Spark CI walltime from ~84 min to ~58 min (−31%)

Ask

One more potential fix is to reduce the runtime of the tests inside spark-ci. I've pulled data on how long each test class takes to run. See https://gist.github.com/kevinjqliu/86827bedf9d02adad36881f476484208#top-25--core-spark for more details.
Some of the tests are inherently slow, but exacerbated by parameterized matrix that dramatically multiples the number of runs.

Is it worth to dig into these tests and see if there are any optimizations?

[kevinjqliu]

another potential optimization is to minimize the number of triggers for spark-ci. ie

• flink changes dont need to trigger spark-ci
• spark 3.5 changes dont need to trigger spark 4.x
• etc
  we should be careful here and only apply optimizations when absolutely sure

this will reduce the number of jobs, but the walltime is still dominated by a single matrix run (which after #16357 should be around 58 mins)

removing spark 3.4 is the same flavor of change