Spark 4.0 - Duplicate and Adjust code base from previous versions

[BentsiLeviav]

All versions' implementation in this connector shares the same code base, with some adjustments for each version, for instance:

• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/ClickHouseSQLConf.scala
• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/ExprUtils.scala
• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/JsonWriter.scala
• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SchemaUtils.scala (Spark 4.0 - Add VARIANT data type support #404 Writing StructType to ClickHouse #278)
• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkOptions.scala
• spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkUtils.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/ClickHouseCatalog.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/ClickHouseCommandRunner.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/ClickHouseHelper.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/ClickHouseTable.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/CommitMessage.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/Constants.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/Metrics.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/SQLHelper.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/CityHash64.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/FunctionRegistry.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/MultiStringArgsHash.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/MurmurHash2.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/MurmurHash3.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/func/XxHash64.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/ClickHouseMetadataColumn.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/ClickHouseRead.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/ClickHouseReader.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/InputPartitions.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/ScanJobDescription.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/format/ClickHouseBinaryReader.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/read/format/ClickHouseJsonReader.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/write/ClickHouseWrite.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/write/ClickHouseWriter.scala
• spark-3.5/clickhouse-spark/src/main/scala/com/clickhouse/write/WriteJobDescription.scala
• spark-3.5/clickhouse-spark/src/main/scala/xenon/clickhouse/write/format/ClickHouseArrowStreamWriter.scala
• spark-3.5/clickhouse-spark/src/main/scala/xenon/clickhouse/write/format/ClickHouseJsonEachRowWriter.scala

This is a partial list, and there might be more files to copy & adjust.

[pan3793]

I would suggest not doing that unless it causes real trouble.

The current code structure is inspired by Apache Iceberg. Spark is evolving fast, some internal APIs used by the connector may have breaking changes across versions, duplicating code makes us always focus on the latest Spark version and don't need to struggle with runtime linkage issues.

[BentsiLeviav]

@pan3793

We will adjust the code to work with Spark 4.0.0 and not just copy and paste it, but do you happen to have some specific examples in mind?

[pan3793]

Oh, I might have misunderstood this ticket. I was thinking you want to avoid code duplication.

Copy the whole spark-3.5 to spark-4.0, then adjust the code and build script is a simple way to support Spark 4.0

[pan3793]

If you want to keep a clear diff in the commit history, you can make an as-is copy and merge it to the main branch first, then make changes to pass CI

[BentsiLeviav]

@ShimonSte, can this be closed?

[ShimonSte]

Yes, was done in #452