[Improvement]: cherry-pick some issue to 0.4.x

[Aireed]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Because there are still users staying on version 0.4.x, we need to ensure that bugs are merged into the 0.4.x branch.

How should we improve?

Compare the 0.4.x branch and the master branch to identify which pull requests need to be merged into the 0.4.x branch, and confirm with the relevant person in charge.

Flink

• [ARCTIC-1283][FLINK]when source is a kafka with canal-json format, the task can't run with a exception thrown #1315
• [ARCTIC-1306][FLINK] Arctic Define watermark on unkeyed table encounters FieldNotFound Exception #1312
• [Arctic-414][Improvement]: Refresh the arctic properties in the FlinkTablePropertiesInvocationHandler after table refreshed #1393

spark

• [ARCTIC-1040][SPARK][Improvement]Refactor code and using RowLevelWrite operator to implement upsert/delete/update #1170
• [ARCTIC-1346][AMS]Fix bug: When AMS is deployed HA, use Kerberos to access non-Kerberos zookeeper will throw an exception #1414
• [ARCTIC-1404][Core]Fix bug: Unable to synchronize newly added fields in non-primary table to Hive when use Spark #1405
• [ARCTIC-1369][Hive]Fix bug: AlterTableAddColumn not support field with uppercase letters in Spark #1370
• [ARCTIC-1346][Core]: Clear kerberos ticket to access zk #1345
• [Arctic-1365]: Remove the code of spark 2.3 module #1381

optimize

• [ARCTIC-1367][hive] fix NullPointerException for null partition value for Mixed Hive Table #1371
• [ARCTIC-1380] [core] fix NPE when scan entries of Iceberg table < 1.0.0 #1396
• [ARCTIC-1394] Optimize the execution time of drop table #1395
• [ARCTIC-725] [optimizer,ams]FlinkOptimizer automatically retries after failover  #1248

other

• [Arctic-1318][Improvement]: we should throw a exception when we have no privilege to access the files of table #1376
• [Arctic-1120]we return bytes array when we response file load request #1379
• [Arctic-1347][Core]The position-delete written by iceberg directly through the base table cannot take effect when read by Arctic #1348
• [Arctic-1349][Core]Reuse Spark SparkStructLike and Container and SparkInternalRowWrapper #1351

do not cherry-pick

• [ARCTIC-1308] The front-end ams Settings page should not display quotes #1309
• [Arctic-1266][AMS]: Use ArcticSparkSessionCatalog for terminal #1264
• [ARCTIC-1059][Spark][Improvement]: Add scala code style check for spark module #1133
• [Arctic-506]:Upgrade iceberg version to 1.1.x #989
• [ARCTIC-1327]Codecov Report shows coverage is 0.00% #1328
• [Arctic-1364]: Disable jacoco in github CI #1355
• [ARCTIC-965][AMS] Fix temp method use ArcticTableOperations for Kerberos auth #984
• [ARCTIC-261][Spark]Support Spark 3.3 #1259
• [Arctic-1120]we return bytes array when we response file load request #1379
• [ARCTIC-1383][Spark]Support spark 3.2 #1389
• [ARCTIC-1382]Improve unit test codes in module Hive #1416
• [ARCTIC-1313] [doc] fix default value of self-optimizing.major.trigger.duplicate-ratio #1314
• [ARCTIC-1401][Improvement]: Arctic Flink needs to organize the community-related code into a separate module #1413 for code dif。
• [ARCTIC-1431][FLINK][Bug]: when read from Logstore, will encounter ClassCastException #1432 merged
• [ARCTIC-1319] [ams] Major Self-Optimizing should not delete pos-delete files #1320 merged
• [ARCTIC-1288][ams] limit the total files size bytes for Optimizing #1294 merged
• [ARCTIC-1386][ams] fix Iceberg Format table's orphan files clean mistakenly delete the equality-delete files  #1390 no need merge
• [ARCTIC-1388][Spark]Fix bug: The runtime jar of Spark 3.3 throws a NoClassDefFoundError when executing SQL. #1391 no need merge spark3.3 bug
• [Arctic-1397]: Fix bug: Hive function can't be used when using ArcticSparkSessionCat… #1398 no need merge spark3.3 bug

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[Aireed]

done