[brach-2.0](routine-load) fix consume data too slow in partial partitions

[sollhui]

Proposed changes

#32126

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

[sollhui]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TPC-H: Total hot run time: 49373 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e24613831063cf0befa636c0f31b57e6b970847, data reload: false

------ Round 1 ----------------------------------
q1	18003	4441	4362	4362
q2	2039	145	136	136
q3	10296	1838	1932	1838
q4	10365	1231	1294	1231
q5	8436	4119	3872	3872
q6	233	122	123	122
q7	2055	1571	1586	1571
q8	9512	2693	2696	2693
q9	13364	10423	10433	10423
q10	8643	3521	3470	3470
q11	418	234	244	234
q12	457	297	301	297
q13	18325	3950	4048	3950
q14	349	321	321	321
q15	498	459	442	442
q16	696	590	578	578
q17	1114	954	915	915
q18	7315	6729	6965	6729
q19	1665	1516	1464	1464
q20	530	296	289	289
q21	4381	4056	4046	4046
q22	498	396	390	390
Total cold run time: 119192 ms
Total hot run time: 49373 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4315	4315	4315
q2	315	223	216	216
q3	4151	4142	4182	4142
q4	2758	2756	2765	2756
q5	7324	7209	7200	7200
q6	233	118	118	118
q7	3283	2908	2816	2816
q8	4302	4451	4516	4451
q9	17163	17044	17095	17044
q10	4253	4381	4425	4381
q11	737	680	689	680
q12	1062	923	909	909
q13	7327	3867	3884	3867
q14	477	442	439	439
q15	489	458	464	458
q16	786	739	740	739
q17	3874	3877	3756	3756
q18	9129	8851	8709	8709
q19	1692	1687	1660	1660
q20	2359	2105	2149	2105
q21	8503	8480	8480	8480
q22	994	918	901	901
Total cold run time: 85560 ms
Total hot run time: 80142 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.81% (8039/21263)
Line Coverage: 29.48% (65606/222526)
Region Coverage: 28.95% (33757/116599)
Branch Coverage: 24.79% (17325/69874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e24613831063cf0befa636c0f31b57e6b970847_5e24613831063cf0befa636c0f31b57e6b970847/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 200066 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e24613831063cf0befa636c0f31b57e6b970847, data reload: false

query1	900	395	379	379
query2	6529	2212	2024	2024
query3	6920	197	195	195
query4	20144	17861	17850	17850
query5	19717	6491	6453	6453
query6	314	209	226	209
query7	4423	291	293	291
query8	266	239	222	222
query9	3096	2692	2606	2606
query10	440	282	278	278
query11	11256	10658	10717	10658
query12	121	75	73	73
query13	5581	643	626	626
query14	18342	12940	13452	12940
query15	378	228	238	228
query16	6445	271	261	261
query17	1469	1476	877	877
query18	2266	410	391	391
query19	196	142	148	142
query20	76	76	71	71
query21	184	92	93	92
query22	5318	5015	5139	5015
query23	32355	31744	31761	31744
query24	6690	6536	6438	6438
query25	512	426	420	420
query26	500	154	163	154
query27	1809	294	294	294
query28	6056	2206	2166	2166
query29	2899	2693	2838	2693
query30	239	158	156	156
query31	896	784	711	711
query32	67	57	57	57
query33	391	248	238	238
query34	835	455	478	455
query35	1082	887	872	872
query36	1573	1408	1527	1408
query37	87	62	57	57
query38	3112	2909	2866	2866
query39	1356	1346	1315	1315
query40	207	96	94	94
query41	35	32	33	32
query42	81	85	84	84
query43	663	591	599	591
query44	1118	710	701	701
query45	237	225	223	223
query46	1231	975	953	953
query47	1908	1662	1932	1662
query48	976	683	661	661
query49	612	369	363	363
query50	866	605	570	570
query51	4755	4636	4693	4636
query52	81	78	74	74
query53	442	316	309	309
query54	2646	2440	2444	2440
query55	86	71	77	71
query56	191	183	210	183
query57	1185	1165	1305	1165
query58	204	202	197	197
query59	3325	3121	3158	3121
query60	194	179	195	179
query61	81	77	79	77
query62	873	448	454	448
query63	462	339	333	333
query64	2315	1474	1427	1427
query65	3582	3554	3535	3535
query66	753	369	353	353
query67	15992	15733	14888	14888
query68	10164	641	648	641
query69	567	324	328	324
query70	2696	1596	1688	1596
query71	422	305	295	295
query72	6551	3466	3382	3382
query73	730	318	318	318
query74	6370	5937	5874	5874
query75	5216	3615	3798	3615
query76	6589	1194	1228	1194
query77	1158	238	236	236
query78	12491	11656	12255	11656
query79	8907	622	635	622
query80	969	400	389	389
query81	490	230	235	230
query82	1150	101	95	95
query83	156	132	125	125
query84	257	69	65	65
query85	833	276	272	272
query86	408	362	394	362
query87	3240	3041	3047	3041
query88	4257	2306	2307	2306
query89	405	310	277	277
query90	1988	208	204	204
query91	157	119	122	119
query92	56	51	53	51
query93	5301	579	604	579
query94	699	197	191	191
query95	1097	1039	1062	1039
query96	642	338	325	325
query97	6467	6326	6380	6326
query98	187	182	172	172
query99	3086	850	1043	850
Total cold run time: 313647 ms
Total hot run time: 200066 ms

[doris-robot]

ClickBench: Total hot run time: 30.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e24613831063cf0befa636c0f31b57e6b970847, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.05
query4	1.82	0.06	0.09
query5	0.53	0.52	0.53
query6	1.30	0.62	0.63
query7	0.01	0.00	0.01
query8	0.04	0.02	0.02
query9	0.53	0.47	0.48
query10	0.54	0.53	0.53
query11	0.12	0.09	0.08
query12	0.12	0.10	0.09
query13	0.62	0.61	0.61
query14	0.81	0.76	0.78
query15	0.78	0.75	0.76
query16	0.39	0.37	0.36
query17	1.01	1.02	1.01
query18	0.23	0.26	0.22
query19	1.92	1.80	1.81
query20	0.02	0.01	0.02
query21	15.45	0.56	0.57
query22	2.37	2.43	1.76
query23	17.28	1.01	0.96
query24	6.92	0.89	0.97
query25	0.39	0.12	0.05
query26	0.67	0.15	0.16
query27	0.05	0.04	0.03
query28	6.44	0.72	0.75
query29	12.63	2.36	2.31
query30	0.58	0.50	0.52
query31	2.80	0.41	0.38
query32	3.37	0.50	0.49
query33	3.06	3.06	3.07
query34	15.26	4.78	4.78
query35	4.87	4.82	4.84
query36	1.05	1.03	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.7 s
Total hot run time: 30.72 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5e24613831063cf0befa636c0f31b57e6b970847 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       19.8 seconds inserted 10000000 Rows, about 505K ops/s

[dataroaring]

LGTM