[VL][CI] Migrate Spark 4.1 tests to CentOS 9

[ReemaAlzaid]

What changes are proposed in this pull request?

This PR adds CentOS 9 support for running Spark unit tests while maintaining existing CentOS 8 jobs for gradual migration.
Related Issue: #11511
Fixes: #11481

How was this patch tested?

Will be tested by CI pipeline execution

[PHILO-HE]

@ReemaAlzaid, thanks for the PR. I'm concerned that this may introduce too much test redundancy, considering these Spark tests are being verified on Centos 8. If the purpose is enabling some Spark tests with python 3.11, can we install this python version by compiling from source to keep using Centos 8? cc @zhouyuan

If we indeed want to migrate from Centos 8 to 9 for Spark tests, maybe we can directly do the replacement. There seems to be no need to keep the both categories. In addition, we may re-use the existing build native lib job, i.e., for tests on Centos 9, download the native binary built on Centos 7. I assume there is no compatibility issue.

[zhouyuan]

@ReemaAlzaid Thanks for the contribution.

[zhouyuan]

@ReemaAlzaid since it may take some time to get the patches ready in velox, could you please check if we could exclude the new timezone(America/Coyhaique) in those tests? here's one example:
https://github.com/apache/incubator-gluten/blob/main/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenTryCastSuite.scala#L40

[zhouyuan]

👍

[PHILO-HE]

Looks good. Thank you.

[baibaichen]

@ReemaAlzaid would you pelase create new pr for this https://github.com/apache/incubator-gluten/pull/11481/changes#diff-d34bc9515f676ce2605c33b00515f94ff1d6b0f6eb1be32670f9783d79c85968