[branch-2.0](routine-load) optimize routine load task schedule to make consume real-time (#31273)

[sollhui]

Proposed changes

pick from: #31273

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[sollhui]

run buildall

[sollhui]

run buildall

[doris-robot]

TPC-H: Total hot run time: 50404 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 327b3acbd7ed595fc25bebb958bf26d0ad3e96e2, data reload: false

------ Round 1 ----------------------------------
q1	18149	4382	4311	4311
q2	2060	153	149	149
q3	10270	1882	1937	1882
q4	10335	1275	1322	1275
q5	8563	3881	4017	3881
q6	234	152	143	143
q7	2037	1587	1613	1587
q8	9321	2747	2724	2724
q9	11164	10600	10497	10497
q10	8683	3511	3538	3511
q11	426	252	243	243
q12	478	302	306	302
q13	18351	4014	4051	4014
q14	358	331	336	331
q15	509	473	460	460
q16	689	560	574	560
q17	1142	981	987	981
q18	7308	6957	7286	6957
q19	1796	1666	1645	1645
q20	580	337	331	331
q21	4562	4181	4187	4181
q22	540	439	464	439
Total cold run time: 117555 ms
Total hot run time: 50404 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4504	4494	4473	4473
q2	348	255	245	245
q3	4272	4194	4145	4145
q4	2757	2798	2764	2764
q5	7316	7185	7212	7185
q6	237	119	117	117
q7	3292	2838	2841	2838
q8	4352	4491	4453	4453
q9	17464	17152	17069	17069
q10	4211	4251	4266	4251
q11	761	713	686	686
q12	1032	839	872	839
q13	7032	3720	3726	3720
q14	459	416	426	416
q15	517	471	455	455
q16	723	688	674	674
q17	3857	3796	3800	3796
q18	8763	8869	8787	8787
q19	1729	1720	1663	1663
q20	2377	2152	2100	2100
q21	8454	8547	8444	8444
q22	1048	986	930	930
Total cold run time: 85505 ms
Total hot run time: 80050 ms

[doris-robot]

TPC-DS: Total hot run time: 203132 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 327b3acbd7ed595fc25bebb958bf26d0ad3e96e2, data reload: false

query1	957	394	415	394
query2	6522	2551	2479	2479
query3	6926	207	206	206
query4	20706	17855	17799	17799
query5	19735	6505	6543	6505
query6	290	230	231	230
query7	4153	314	309	309
query8	412	385	394	385
query9	3128	2660	2597	2597
query10	420	306	336	306
query11	11402	10712	10698	10698
query12	126	76	77	76
query13	5604	726	680	680
query14	17990	13512	13365	13365
query15	369	231	254	231
query16	6768	290	270	270
query17	1744	1450	908	908
query18	2275	425	428	425
query19	216	159	152	152
query20	78	87	80	80
query21	190	94	97	94
query22	5105	5077	5135	5077
query23	32561	31621	31975	31621
query24	6925	6478	6554	6478
query25	526	444	424	424
query26	529	159	163	159
query27	1830	295	298	295
query28	6139	2391	2305	2305
query29	2881	2691	2664	2664
query30	246	169	169	169
query31	934	731	741	731
query32	66	70	70	70
query33	413	260	261	260
query34	850	506	476	476
query35	1184	945	921	921
query36	1546	1214	1137	1137
query37	93	60	60	60
query38	3100	2888	2915	2888
query39	1382	1335	1323	1323
query40	200	94	92	92
query41	46	47	43	43
query42	96	86	84	84
query43	674	836	617	617
query44	1114	715	701	701
query45	241	235	234	234
query46	1229	985	962	962
query47	1876	1666	1716	1666
query48	1014	725	716	716
query49	623	375	380	375
query50	854	623	639	623
query51	4751	4702	4662	4662
query52	93	75	87	75
query53	465	325	324	324
query54	2649	2439	2464	2439
query55	82	82	73	73
query56	229	208	206	206
query57	1187	1146	1057	1057
query58	220	202	203	202
query59	4400	3772	3763	3763
query60	218	223	206	206
query61	99	95	100	95
query62	783	534	557	534
query63	489	349	344	344
query64	2331	1556	1452	1452
query65	3656	3552	3557	3552
query66	778	371	383	371
query67	15533	16622	15769	15769
query68	8666	666	648	648
query69	588	341	345	341
query70	1701	1483	1364	1364
query71	413	308	313	308
query72	6546	3507	3484	3484
query73	740	331	318	318
query74	6273	5792	5838	5792
query75	5166	3714	3621	3621
query76	5120	1145	1150	1145
query77	833	262	252	252
query78	12681	11711	12700	11711
query79	11458	654	635	635
query80	1168	395	410	395
query81	489	236	230	230
query82	1375	99	96	96
query83	162	138	128	128
query84	261	72	71	71
query85	960	341	334	334
query86	336	327	289	289
query87	3171	3048	3015	3015
query88	5023	2330	2313	2313
query89	470	288	292	288
query90	1915	222	210	210
query91	170	138	137	137
query92	57	49	55	49
query93	6127	570	565	565
query94	734	207	212	207
query95	1104	1075	1069	1069
query96	651	331	324	324
query97	6515	6285	6420	6285
query98	196	178	174	174
query99	2812	850	936	850
Total cold run time: 316132 ms
Total hot run time: 203132 ms

[doris-robot]

ClickBench: Total hot run time: 30.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 327b3acbd7ed595fc25bebb958bf26d0ad3e96e2, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.04
query4	1.80	0.06	0.06
query5	0.53	0.52	0.51
query6	1.23	0.61	0.61
query7	0.02	0.01	0.00
query8	0.03	0.03	0.02
query9	0.54	0.50	0.49
query10	0.54	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.64	0.61	0.61
query14	0.79	0.80	0.78
query15	0.78	0.76	0.77
query16	0.36	0.37	0.38
query17	1.00	1.00	1.00
query18	0.23	0.25	0.25
query19	1.96	1.84	1.83
query20	0.01	0.01	0.01
query21	15.47	0.57	0.56
query22	2.12	2.40	1.42
query23	17.20	1.01	0.95
query24	4.89	0.88	1.47
query25	0.32	0.14	0.05
query26	0.63	0.17	0.14
query27	0.04	0.04	0.04
query28	7.89	0.76	0.70
query29	12.63	2.39	2.34
query30	0.62	0.53	0.53
query31	2.81	0.39	0.37
query32	3.38	0.49	0.49
query33	3.09	3.11	3.06
query34	15.28	4.84	4.78
query35	4.89	4.86	4.87
query36	1.06	1.03	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.74 s
Total hot run time: 30.52 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 327b3acbd7ed595fc25bebb958bf26d0ad3e96e2 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s