[fix](routine-load) dealing with the high watermark of Kafka may fallback (#35901)

[sollhui]

pick (#35901)

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[sollhui]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49676 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 88fb51b5436a53dbe73a581c924e3ce7a8d4a8ca, data reload: false

------ Round 1 ----------------------------------
q1	18027	4388	4322	4322
q2	2062	152	150	150
q3	10464	1856	1914	1856
q4	10290	1229	1319	1229
q5	8762	4063	3929	3929
q6	233	148	126	126
q7	2021	1578	1596	1578
q8	9310	2718	2696	2696
q9	11024	10691	10284	10284
q10	8691	3504	3476	3476
q11	430	254	244	244
q12	482	297	301	297
q13	18349	3942	4008	3942
q14	349	321	331	321
q15	506	450	467	450
q16	674	569	570	569
q17	1137	973	967	967
q18	7263	7020	6842	6842
q19	1761	1681	1614	1614
q20	542	303	324	303
q21	4445	4047	4075	4047
q22	519	434	445	434
Total cold run time: 117341 ms
Total hot run time: 49676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4317	4272	4318	4272
q2	316	237	230	230
q3	4157	4149	4153	4149
q4	2761	2736	2763	2736
q5	7216	7141	7163	7141
q6	234	121	120	120
q7	3239	2886	2854	2854
q8	4352	4475	4492	4475
q9	17367	17089	17110	17089
q10	4209	4259	4293	4259
q11	774	673	683	673
q12	1038	841	864	841
q13	6923	3734	3753	3734
q14	452	422	421	421
q15	497	460	456	456
q16	733	687	673	673
q17	3841	3864	3844	3844
q18	8724	8753	8839	8753
q19	1707	1727	1685	1685
q20	2395	2133	2110	2110
q21	8537	8392	8489	8392
q22	1025	1011	985	985
Total cold run time: 84814 ms
Total hot run time: 79892 ms

[doris-robot]

TPC-DS: Total hot run time: 203698 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 88fb51b5436a53dbe73a581c924e3ce7a8d4a8ca, data reload: false

query1	936	419	375	375
query2	6546	2818	2711	2711
query3	6913	206	202	202
query4	20963	17971	18025	17971
query5	19736	6533	6553	6533
query6	282	226	232	226
query7	4158	289	293	289
query8	412	368	359	359
query9	3117	2679	2597	2597
query10	431	296	285	285
query11	11238	10715	10687	10687
query12	117	80	73	73
query13	5605	715	679	679
query14	18187	13065	13300	13065
query15	397	241	237	237
query16	6453	286	265	265
query17	1710	1431	882	882
query18	2307	422	401	401
query19	213	153	155	153
query20	78	83	80	80
query21	193	92	90	90
query22	5223	5122	5094	5094
query23	32525	31989	32051	31989
query24	6894	6498	6517	6498
query25	521	430	423	423
query26	526	159	160	159
query27	1839	287	294	287
query28	6218	2385	2322	2322
query29	2994	2615	2678	2615
query30	246	166	169	166
query31	908	764	749	749
query32	67	67	62	62
query33	392	258	256	256
query34	853	469	487	469
query35	1153	976	929	929
query36	1441	1169	1274	1169
query37	90	61	62	61
query38	3105	2954	2962	2954
query39	1376	1311	1307	1307
query40	209	98	96	96
query41	48	44	43	43
query42	84	83	86	83
query43	646	629	647	629
query44	1124	713	715	713
query45	244	235	236	235
query46	1234	981	974	974
query47	1800	1687	1673	1673
query48	1016	715	698	698
query49	624	369	384	369
query50	878	620	632	620
query51	4787	4688	4648	4648
query52	94	78	82	78
query53	452	330	316	316
query54	2643	2447	2443	2443
query55	83	73	72	72
query56	249	230	214	214
query57	1137	1071	1141	1071
query58	213	208	198	198
query59	4232	3947	3887	3887
query60	205	204	198	198
query61	98	95	96	95
query62	800	481	448	448
query63	492	345	339	339
query64	2511	1535	1485	1485
query65	3637	3563	3547	3547
query66	780	379	375	375
query67	15838	15637	15898	15637
query68	8431	654	641	641
query69	562	342	351	342
query70	1670	1373	1341	1341
query71	406	309	312	309
query72	6487	3470	3508	3470
query73	736	319	324	319
query74	6268	5899	5850	5850
query75	5208	3786	3700	3700
query76	5125	1152	1189	1152
query77	872	262	248	248
query78	12708	11805	12564	11805
query79	10590	637	627	627
query80	946	404	397	397
query81	492	236	232	232
query82	1560	106	101	101
query83	175	130	135	130
query84	266	71	74	71
query85	887	346	339	339
query86	330	294	279	279
query87	3204	3089	3004	3004
query88	4764	2299	2290	2290
query89	477	296	285	285
query90	1972	217	208	208
query91	171	140	135	135
query92	61	53	55	53
query93	6495	532	601	532
query94	722	217	209	209
query95	1094	1036	1050	1036
query96	648	321	320	320
query97	6565	6443	6402	6402
query98	188	182	178	178
query99	2826	840	880	840
Total cold run time: 315427 ms
Total hot run time: 203698 ms

[doris-robot]

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 88fb51b5436a53dbe73a581c924e3ce7a8d4a8ca, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.26	0.04	0.05
query4	1.78	0.06	0.06
query5	0.53	0.53	0.52
query6	1.23	0.67	0.62
query7	0.01	0.01	0.00
query8	0.04	0.03	0.02
query9	0.52	0.47	0.48
query10	0.55	0.54	0.53
query11	0.12	0.09	0.08
query12	0.12	0.09	0.10
query13	0.61	0.61	0.62
query14	0.78	0.77	0.78
query15	0.80	0.76	0.75
query16	0.37	0.37	0.36
query17	1.00	0.96	1.01
query18	0.21	0.26	0.26
query19	1.92	1.86	1.83
query20	0.02	0.01	0.01
query21	15.47	0.56	0.56
query22	2.06	1.81	1.48
query23	17.12	1.00	1.09
query24	6.28	1.58	1.01
query25	0.41	0.12	0.05
query26	0.72	0.17	0.14
query27	0.04	0.03	0.04
query28	6.26	0.73	0.70
query29	12.71	2.36	2.31
query30	0.59	0.52	0.52
query31	2.80	0.38	0.37
query32	3.39	0.50	0.48
query33	3.06	3.09	3.12
query34	15.26	4.83	4.80
query35	4.90	4.85	4.85
query36	1.04	1.02	1.01
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.06	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.45 s
Total hot run time: 30.66 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 88fb51b5436a53dbe73a581c924e3ce7a8d4a8ca with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s