Skip to content

branch-3.1: [chore](job) display sequence column in show routine load #53441 #53505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 19, 2025
Merged

branch-3.1: [chore](job) display sequence column in show routine load #53441 #53505

morrySnow merged 1 commit into from
Jul 19, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 18, 2025

Cherry-picked from #53441

@sollhui
[chore](job) display sequence column in show routine load (#53441)
aefac68 
### What problem does this PR solve?

create routine load appoint sequence column with `ORDER BY`
```
CREATE ROUTINE LOAD testShow ON test_show_routine_load
COLUMNS TERMINATED BY ",",
ORDER BY k1
FROM KAFKA
(
    "kafka_broker_list" = "10.16.10.7:19092",
    "kafka_topic" = "test_show_routine_load",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```

show routine load can not display sequence column.

The pr fix the issue and result is:
```
routine load job properties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","column_separator":"','","line_delimiter":"\n","delete":"*","current_concurrent_number":"0","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval":"60","max_batch_size":"1073741824","escape":"\u0000","enclose":"\u0000","partitions":"*","columnToColumnExpr":"","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.0","sequence_col":"k1"}
```
@github-actions github-actions bot requested a review from morrySnow as a code owner July 18, 2025 02:08
@morrySnow morrySnow closed this Jul 18, 2025
@morrySnow morrySnow reopened this Jul 18, 2025
@morrySnow
Copy link
Contributor

morrySnow commented Jul 18, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 18, 2025

FE UT Coverage Report

Increment line coverage 50.00% (2/4) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

TPC-H: Total hot run time: 39859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aefac6864c77f2c66701e137acf31abd5a9a5fb0, data reload: false

------ Round 1 ----------------------------------
q1	17863	6981	6670	6670
q2	2071	194	168	168
q3	10612	1162	1241	1162
q4	10474	744	704	704
q5	7737	2896	2816	2816
q6	224	138	142	138
q7	980	624	598	598
q8	9353	1991	2045	1991
q9	6629	6404	6479	6404
q10	7070	2268	2351	2268
q11	480	259	259	259
q12	400	214	209	209
q13	17790	3023	3011	3011
q14	227	209	203	203
q15	508	462	474	462
q16	473	380	371	371
q17	978	597	520	520
q18	7501	6625	6627	6625
q19	1356	1058	964	964
q20	478	204	202	202
q21	4029	3137	3190	3137
q22	1091	1013	977	977
Total cold run time: 108324 ms
Total hot run time: 39859 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6692	6569	6642	6569
q2	326	227	234	227
q3	2964	3031	2930	2930
q4	2121	1962	1814	1814
q5	5737	5769	5738	5738
q6	223	133	133	133
q7	2221	1823	1786	1786
q8	3377	3548	3558	3548
q9	8724	8882	8811	8811
q10	3610	3538	3505	3505
q11	598	499	511	499
q12	814	603	608	603
q13	9013	3211	3144	3144
q14	295	264	290	264
q15	512	491	478	478
q16	503	455	453	453
q17	1891	1649	1627	1627
q18	8303	7911	7743	7743
q19	1669	1571	1654	1571
q20	2106	1860	1874	1860
q21	5147	5096	5084	5084
q22	1130	1046	1029	1029
Total cold run time: 67976 ms
Total hot run time: 59416 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

TPC-DS: Total hot run time: 198001 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aefac6864c77f2c66701e137acf31abd5a9a5fb0, data reload: false

query1	1292	924	908	908
query2	6297	1916	1884	1884
query3	10960	4631	4557	4557
query4	33543	24014	23991	23991
query5	4814	483	456	456
query6	272	173	187	173
query7	4019	313	329	313
query8	275	234	235	234
query9	9571	2598	2563	2563
query10	484	258	268	258
query11	18176	15509	15601	15509
query12	161	114	109	109
query13	1554	439	432	432
query14	9671	7044	7372	7044
query15	242	186	188	186
query16	7969	545	505	505
query17	1642	619	618	618
query18	2126	346	318	318
query19	247	166	173	166
query20	125	116	118	116
query21	218	116	112	112
query22	4807	4389	4509	4389
query23	34947	33947	34012	33947
query24	11258	2936	3002	2936
query25	635	441	448	441
query26	739	173	171	171
query27	2146	357	380	357
query28	6355	2196	2156	2156
query29	784	476	459	459
query30	265	193	162	162
query31	993	824	814	814
query32	87	55	57	55
query33	657	304	318	304
query34	930	528	540	528
query35	887	745	722	722
query36	1122	944	963	944
query37	112	71	72	71
query38	4066	3930	3967	3930
query39	1523	1468	1507	1468
query40	215	103	111	103
query41	49	48	47	47
query42	119	106	104	104
query43	535	490	476	476
query44	1217	828	815	815
query45	185	172	171	171
query46	1190	761	731	731
query47	1992	1877	1930	1877
query48	435	354	356	354
query49	877	416	417	416
query50	857	445	433	433
query51	7414	7270	7309	7270
query52	112	94	98	94
query53	283	195	184	184
query54	1276	476	488	476
query55	87	82	80	80
query56	269	265	259	259
query57	1295	1218	1183	1183
query58	244	212	213	212
query59	3151	2993	3041	2993
query60	304	271	278	271
query61	106	107	135	107
query62	863	665	725	665
query63	228	195	188	188
query64	3623	696	674	674
query65	3360	3281	3312	3281
query66	922	324	307	307
query67	16163	15585	15634	15585
query68	4428	584	610	584
query69	441	263	264	263
query70	1170	1126	1123	1123
query71	324	307	268	268
query72	6308	3995	3941	3941
query73	773	347	359	347
query74	10103	9230	8901	8901
query75	3393	2615	2668	2615
query76	2757	1047	1115	1047
query77	378	284	281	281
query78	10598	9597	9554	9554
query79	2463	645	629	629
query80	1156	431	444	431
query81	550	219	220	219
query82	905	95	89	89
query83	244	145	146	145
query84	236	77	78	77
query85	1283	307	294	294
query86	444	299	300	299
query87	4347	4308	4168	4168
query88	4114	2440	2420	2420
query89	419	293	293	293
query90	1988	192	191	191
query91	139	110	107	107
query92	64	52	52	52
query93	2292	568	548	548
query94	828	296	318	296
query95	369	264	260	260
query96	629	287	282	282
query97	3311	3173	3108	3108
query98	225	200	202	200
query99	1527	1321	1307	1307
Total cold run time: 301548 ms
Total hot run time: 198001 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

ClickBench: Total hot run time: 30.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aefac6864c77f2c66701e137acf31abd5a9a5fb0, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.54	0.52	0.53
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.02	0.03
query9	0.56	0.50	0.52
query10	0.56	0.54	0.54
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.60	0.60	0.59
query14	0.78	0.78	0.79
query15	0.85	0.85	0.83
query16	0.38	0.36	0.38
query17	1.08	1.08	1.07
query18	0.24	0.22	0.21
query19	1.92	1.88	1.86
query20	0.01	0.01	0.01
query21	15.39	0.57	0.56
query22	2.27	2.50	1.63
query23	17.00	1.06	0.84
query24	3.03	1.53	1.41
query25	0.21	0.26	0.21
query26	0.27	0.14	0.14
query27	0.04	0.04	0.05
query28	9.56	0.50	0.45
query29	12.60	3.26	3.24
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.23	0.46	0.46
query33	2.97	2.96	3.04
query34	17.09	4.50	4.59
query35	4.55	4.58	4.56
query36	0.68	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.43 s
Total hot run time: 30.7 s

@morrySnow morrySnow merged commit 645fdb6 into branch-3.1 Jul 19, 2025
22 checks passed
@morrySnow morrySnow deleted the auto-pick-53441-branch-3.1 branch July 19, 2025 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morrySnow @hello-stephen @doris-robot @sollhui