[fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error

[xiaoDjun]

Proposed changes

Issue Number: close #14600

Problem summary

In spark load, the job failed when the partition column is not in the duplicate key column list.

We traced the sparkDpp.java and found that, the partition columns ware built with the key index in base metadata. But when partition column was not key, an IndexOutOfBoundsException occurs when obtaining it through DppColumns.

The exception is:

java.lang.IndexOutOfBoundsException: Index: 3, Size: 1 
at java.util.ArrayList.rangeCheck(ArrayList.java:657) 
at java.util.ArrayList.get(ArrayList.java:433) 
at org.apache.doris.load.loadv2.dpp.DppColumns.<init>(DppColumns.java:39) 
at org.apache.doris.load.loadv2.dpp.DorisRangePartitioner.getPartition(DorisRangePartitioner.java:56) 
at org.apache.doris.load.loadv2.dpp.SparkDpp$2.call(SparkDpp.java:487) 
at org.apache.doris.load.loadv2.dpp.SparkDpp$2.call(SparkDpp.java:460) 
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) 
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143) 
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) 
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) 
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) 
at org.apache.spark.scheduler.Task.run(Task.scala:109) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748) 
Driver stacktrace:

Checklist(Required)

1. Does it affect the original behavior:
   • Yes
   • No
   • I don't know
2. Has unit tests been added:
   • Yes
   • No
   • No Need
3. Has document been added or modified:
   • Yes
   • No
   • No Need
4. Does it need to update dependencies:
   • Yes
   • No
5. Are there any changes that cannot be rolled back:
   • Yes (If Yes, please explain WHY)
   • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[hf200012]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[hf200012]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[hello-stephen]

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.61 seconds
load time: 486 seconds
storage size: 17123333158 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221129051721_clickbench_pr_54806.html