[Subtask] [spark-connector] support multi spark&scala version

[FANNG1]

Describe the subtask

support spark 2.4 and scala 2.13 for now, we should support multi spark and scala versions

• [Improvement] validate spark connector on spark 3.1.2 #3007
• [Improvement] move spark connector integration test from integration-test module to spark-connector module #3305
• [Improvement] use old create table interface to support mult spark version #3330
• [Bug report] Log4jLoggerFactory package conflict when running on spark 3.3 #3320
• [Subtask] support multi spark version #3554
• [Subtask] support multi scala version #3555

Parent issue

#1227

[caican00]

Hi @FANNG1 will we consider support for spark3.1 and spark3.3 latter? Currently the main versions of spark in XIAOMI are spark3.1 and spark3.3.

[FANNG1]

yes, but we planned to do this after the basic feature of the spark-connector is set up. If you're interested, you could do some work.

[caican00]

yes, but we planned to do this after the basic feature of the spark-connector is set up. If you're interested, you could do some work.

It sounds reasonable. Let's set up the basic feature first and if there are version compatibility issues, we can build on a lower version first in XIAOMI.