[GLUTEN-8889][CORE] Bump Spark version from 3.5.2 to 3.5.5

[jackylee-ch]

What changes were proposed in this pull request?

Support Spark 3.5.5 and upgrade iceberg version to 1.8.1 to fix the compatibility issue with Spark 3.5.4.

How was this patch tested?

GA

[github-actions]

#8889

[github-actions]

Run Gluten ClickHouse CI on ARM

[baibaichen]

@jlfsdtc please fix spark 35 pipeline of ch

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[baibaichen]

@jackylee-ch

Let's first upgrade to 3.5.5 and then upgrade Icerberg? After upgrading to 1.8.1, Icerberg only supports JDK 17, and clickhouse backend need take some time to support.

@jlfsdtc will open pr for upgradeing to 3.5.5.

[jackylee-ch]

Let's first upgrade to 3.5.5 and then upgrade Icerberg? After upgrading to 1.8.1, Icerberg only supports JDK 17, and clickhouse backend need take some time to support.

@baibaichen We need upgrade them both, as Iceberg 1.8.1 fix the core_dump problem with Spark 3.5.4.

[baibaichen]

Let's first upgrade to 3.5.5 and then upgrade Icerberg? After upgrading to 1.8.1, Icerberg only supports JDK 17, and clickhouse backend need take some time to support.

@baibaichen We need upgrade them both, as Iceberg 1.8.1 fix the core_dump problem with Spark 3.5.4.

I understand this，We will try to support JDK 17 in the next month. My suggestion is to upgrade to 3.5.5 first, then use different iceberg versions for different JDKs, and finally, Clickhouse support JDK 17.

Some users are still using JDK 8, and we still need to support them.

[jackylee-ch]

I understand this，We will try to support JDK 17 in the next month. My suggestion is to upgrade to 3.5.5 first, then use different iceberg versions for different JDKs, and finally, Clickhouse support JDK 17.

Some users are still using JDK 8, and we still need to support them.

It's a block issue for iceberg with Spark 3.5.5. To upgrade to Spark 3.5.5, we must use Iceberg 1.8.1.

[github-actions]

Run Gluten ClickHouse CI on ARM

[zhouyuan]

should we also enable jdk17 here?

[jackylee-ch]

We can’t define JDK 17 here. If we define the java.version to 17 in the profile spark-3.5, then mvn -Pspark-3.5 -Pjava-8 and mvn -Pjava-8 -Pspark-3.5 would exhibit different behaviors.

[zhouyuan]

I understand this，We will try to support JDK 17 in the next month. My suggestion is to upgrade to 3.5.5 first, then use different iceberg versions for different JDKs, and finally, Clickhouse support JDK 17.
Some users are still using JDK 8, and we still need to support them.

It's a block issue for iceberg with Spark 3.5.5. To upgrade to Spark 3.5.5, we must use Iceberg 1.8.1.

Agreed we better to upgrade on both - unless we drop the iceberg profile on Spark 3.5.5 firstly @baibaichen

[baibaichen]

Agreed we better to upgrade on both - unless we drop the iceberg profile on Spark 3.5.5 firstly @baibaichen

OK

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[zhouyuan]

CK backend added JDk17 support
#9242

[jackylee-ch]

@zzcclp I’m still encountering issues with JDK 8.

[zzcclp]

@zzcclp I’m still encountering issues with JDK 8.

There is jdk version issue when using JDK 8

[zzcclp]

Run Gluten Clickhouse CI on x86

[zzcclp]

The GlutenBroadcastJoinSuite is hanging, the logs are below:

Please disable this suite first and then run again?

The GlutenSortShuffleSuite also hang, it needs disable first.

[jackylee-ch]

Spark 3.5.5 UT for CH backends has been passed.

[zzcclp]

The latest CI will be failed, the GlutenSortShuffleSuite also hang

[github-actions]

Run Gluten ClickHouse CI on ARM

[jackylee-ch]

The latest CI will be failed, the GlutenSortShuffleSuite also hang

OK, I have disabled this UT to check if it has other problems.

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten ClickHouse CI on ARM

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[zzcclp]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten ClickHouse CI on ARM

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten ClickHouse CI on ARM

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten ClickHouse CI on ARM

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[jackylee-ch]

@zhztheplayer @zzcclp UT has been passed expect some Suites/cases are ignored.

[zhztheplayer]

Fantastic! @jackylee-ch Thank you for taking care of this millstone constantly in the past few months.

Thank you for all the kindly help with the CH CI! @zzcclp

Without your effort we are going to stay on 3.5.2 permanently :)