[CORE] Support Spark 4.0 and Deprecate JDK 8

[weiting-chen]

Description

Current Spark and JDK support status are in below
Spark 3.4.4 support JDK8(8u362) or above version
Spark 3.5.4 support JDK8(8u371) or above version
Spark 4.0 can only support JDK 11 and 17
For better CI resource usage, we can consider to deprecate JDK 8 support when adding Spark 4.0 support.
The plan to support Spark 4.0 will be initialized in Gluten 1.4 and done the work in Gluten 1.5.

One other consideration is in Iceberg cannot support JDK 8 with Spark 3.4.4, we have Iceberg UT may consider to separate the UT testing for Spark 3.4.4 and Iceberg or just support JDK 11/17 for better alignment.

[Yohahaha]

would you send a email to gluten dev mail list to notify all developers?

[weiting-chen]

Yes, will send a mail to dev mail list.

[FelixYBW]

@weiting-chen we may move our CI to JDK17 now but still leave the legacy jdk8 support code there, we still accept PRs for JDK8. Customer who are using jdk8 needs to run the CI from their internal env.

Once we release Gluten1.4, we may remove the JDK8 support completely.

[jackylee-ch]

Once we release Gluten1.4, we may remove the JDK8 support completely.

This means we would drop Spark 3.2 and Spark 3.3 support after Gluten 1.4?

[zhztheplayer]

Is it possible to build Gluten with higher JDK version only for -P spark-4.0?

[xumingming]

Can we keep the support for Spark 3.3?