Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[refactor](jdbc catalog) split clickhouse jdbc executor #34794

Merged
merged 2 commits into from
May 16, 2024
Merged

[refactor](jdbc catalog) split clickhouse jdbc executor #34794

merged 2 commits into from
May 16, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented May 13, 2024

Proposed changes

Issue Number: close #xxx

Split ClickHouseJdbcExecutor for better compatibility

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented May 13, 2024

run buildall

@zy-kkk zy-kkk mentioned this pull request May 13, 2024
Closed
12 tasks
@doris-robot
Copy link

TPC-H: Total hot run time: 40534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 055fe7a130eb43e12f92ec56637317f45f2f1e4a, data reload: false

------ Round 1 ----------------------------------
q1	6801	4281	4280	4280
q2	659	181	192	181
q3	2362	1165	1214	1165
q4	1018	713	789	713
q5	2670	2672	2624	2624
q6	220	134	135	134
q7	1008	589	600	589
q8	2039	2365	2188	2188
q9	6868	6823	6706	6706
q10	4126	3882	3942	3882
q11	360	249	254	249
q12	402	227	223	223
q13	16810	3155	3146	3146
q14	263	217	209	209
q15	500	480	461	461
q16	486	395	366	366
q17	979	656	603	603
q18	7934	7514	7431	7431
q19	1591	1508	1508	1508
q20	542	305	314	305
q21	5003	3285	4002	3285
q22	352	286	286	286
Total cold run time: 62993 ms
Total hot run time: 40534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4264	4244	4232	4232
q2	369	260	254	254
q3	3053	2757	2701	2701
q4	1833	1566	1583	1566
q5	5289	5267	5267	5267
q6	213	124	122	122
q7	2216	1878	1820	1820
q8	3248	3344	3367	3344
q9	8422	8349	8381	8349
q10	3898	3653	3674	3653
q11	582	490	499	490
q12	784	587	621	587
q13	15849	2960	2951	2951
q14	286	258	254	254
q15	513	493	484	484
q16	467	406	425	406
q17	1796	1514	1471	1471
q18	7634	7611	7376	7376
q19	1677	1572	1586	1572
q20	1982	1790	1769	1769
q21	4940	4700	4774	4700
q22	564	486	505	486
Total cold run time: 69879 ms
Total hot run time: 53854 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185858 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 055fe7a130eb43e12f92ec56637317f45f2f1e4a, data reload: false

query1	905	380	358	358
query2	6450	2493	2369	2369
query3	6647	212	214	212
query4	23195	21151	21251	21151
query5	4179	419	423	419
query6	257	180	176	176
query7	4586	291	284	284
query8	242	199	191	191
query9	8525	2425	2385	2385
query10	439	254	263	254
query11	14910	14114	14098	14098
query12	136	91	89	89
query13	1672	381	365	365
query14	9803	6871	7657	6871
query15	217	167	174	167
query16	7789	269	264	264
query17	1814	584	547	547
query18	1870	277	270	270
query19	194	158	149	149
query20	91	86	86	86
query21	194	131	125	125
query22	5016	4820	4789	4789
query23	34084	33491	33357	33357
query24	7194	2968	2905	2905
query25	581	374	375	374
query26	711	155	151	151
query27	2183	313	323	313
query28	4993	2077	2042	2042
query29	854	621	601	601
query30	268	149	155	149
query31	991	765	734	734
query32	87	52	52	52
query33	513	248	249	248
query34	882	476	479	476
query35	762	687	673	673
query36	1053	899	902	899
query37	101	69	65	65
query38	2894	2745	2785	2745
query39	1633	1550	1570	1550
query40	197	129	124	124
query41	41	38	39	38
query42	101	98	96	96
query43	609	561	585	561
query44	1113	731	739	731
query45	263	244	241	241
query46	1064	743	708	708
query47	1979	1901	1919	1901
query48	388	297	290	290
query49	1032	406	397	397
query50	772	388	390	388
query51	6804	6693	6806	6693
query52	103	89	89	89
query53	346	291	278	278
query54	536	432	449	432
query55	76	73	71	71
query56	242	217	217	217
query57	1228	1120	1150	1120
query58	228	206	200	200
query59	3777	3410	3210	3210
query60	260	233	234	233
query61	140	108	90	90
query62	640	460	478	460
query63	311	285	285	285
query64	8481	7412	7365	7365
query65	3156	3118	3126	3118
query66	813	339	338	338
query67	15423	15076	14749	14749
query68	4499	539	535	535
query69	488	301	310	301
query70	1118	1129	1118	1118
query71	369	272	264	264
query72	7510	2586	2436	2436
query73	706	322	318	318
query74	6564	6151	6092	6092
query75	3333	2686	2591	2591
query76	2421	978	978	978
query77	464	263	260	260
query78	10579	10120	10024	10024
query79	1464	524	504	504
query80	1111	438	444	438
query81	520	221	227	221
query82	945	96	94	94
query83	248	166	164	164
query84	239	86	85	85
query85	1093	267	266	266
query86	379	316	327	316
query87	3292	3079	3136	3079
query88	3011	2321	2338	2321
query89	459	377	392	377
query90	1906	190	191	190
query91	126	98	102	98
query92	55	49	50	49
query93	1110	516	505	505
query94	1193	184	189	184
query95	393	303	316	303
query96	578	264	265	264
query97	3180	3000	3000	3000
query98	243	216	213	213
query99	1208	907	894	894
Total cold run time: 269842 ms
Total hot run time: 185858 ms

@zy-kkk
Copy link
Member Author

zy-kkk commented May 14, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41616 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd6ba1355e9169ce7b7daf3bcb3a2e11e7fcd5fe, data reload: false

------ Round 1 ----------------------------------
q1	17609	4249	4208	4208
q2	2014	191	187	187
q3	10483	1205	1198	1198
q4	10194	771	840	771
q5	7445	2655	2708	2655
q6	214	132	136	132
q7	1015	598	579	579
q8	9229	2142	2060	2060
q9	9327	6719	6700	6700
q10	9758	3877	3897	3877
q11	453	238	243	238
q12	473	224	224	224
q13	17450	3167	3211	3167
q14	257	215	204	204
q15	498	462	473	462
q16	476	387	396	387
q17	976	771	715	715
q18	8326	7783	7779	7779
q19	5708	1577	1510	1510
q20	642	317	322	317
q21	5254	4140	3971	3971
q22	347	288	275	275
Total cold run time: 118148 ms
Total hot run time: 41616 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4492	4403	4366	4366
q2	374	280	274	274
q3	3177	2971	2754	2754
q4	1887	1603	1628	1603
q5	5510	5487	5480	5480
q6	213	122	124	122
q7	2347	1980	1983	1980
q8	3254	3362	3402	3362
q9	8618	8670	8561	8561
q10	3922	3806	3819	3806
q11	599	505	496	496
q12	797	582	591	582
q13	15909	3144	3185	3144
q14	315	273	276	273
q15	523	483	489	483
q16	490	434	419	419
q17	1758	1473	1457	1457
q18	7690	7647	7536	7536
q19	1655	1544	1553	1544
q20	1967	1753	1757	1753
q21	9512	4752	4717	4717
q22	584	468	484	468
Total cold run time: 75593 ms
Total hot run time: 55180 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186956 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd6ba1355e9169ce7b7daf3bcb3a2e11e7fcd5fe, data reload: false

query1	916	376	347	347
query2	6437	2440	2252	2252
query3	6647	206	206	206
query4	23130	21214	21392	21214
query5	4142	406	414	406
query6	258	175	175	175
query7	4578	292	282	282
query8	241	193	199	193
query9	8486	2442	2403	2403
query10	432	242	243	242
query11	15041	14234	14413	14234
query12	141	95	84	84
query13	1635	375	365	365
query14	10448	8381	8213	8213
query15	210	182	175	175
query16	7730	260	251	251
query17	1799	548	540	540
query18	1952	268	261	261
query19	208	146	150	146
query20	90	87	89	87
query21	188	132	134	132
query22	5025	4948	4775	4775
query23	34252	33534	33448	33448
query24	6748	2960	2829	2829
query25	532	381	380	380
query26	699	157	152	152
query27	1908	324	323	323
query28	3828	2070	2048	2048
query29	875	630	618	618
query30	237	156	152	152
query31	960	764	792	764
query32	74	52	52	52
query33	487	247	239	239
query34	869	476	475	475
query35	753	675	692	675
query36	1060	904	893	893
query37	101	66	64	64
query38	2906	2757	2747	2747
query39	1611	1536	1589	1536
query40	199	124	124	124
query41	40	40	38	38
query42	101	93	94	93
query43	601	549	574	549
query44	1084	719	729	719
query45	258	249	261	249
query46	1058	755	706	706
query47	2008	1885	1869	1869
query48	384	314	299	299
query49	760	386	377	377
query50	742	398	376	376
query51	6805	6797	6755	6755
query52	98	90	93	90
query53	350	273	277	273
query54	527	423	448	423
query55	74	71	72	71
query56	252	219	222	219
query57	1264	1170	1170	1170
query58	209	202	201	201
query59	3598	3207	3236	3207
query60	251	225	232	225
query61	88	85	86	85
query62	567	452	469	452
query63	306	275	277	275
query64	7960	7387	7337	7337
query65	3131	3096	3088	3088
query66	779	372	338	338
query67	15435	14835	14974	14835
query68	4551	530	530	530
query69	488	298	294	294
query70	1151	1056	1090	1056
query71	388	274	271	271
query72	7115	2523	2349	2349
query73	703	326	326	326
query74	6539	6003	6018	6003
query75	3327	2613	2612	2612
query76	2282	941	994	941
query77	406	258	300	258
query78	10596	10114	9939	9939
query79	2505	506	514	506
query80	1000	434	425	425
query81	519	215	222	215
query82	748	92	94	92
query83	253	161	161	161
query84	240	86	88	86
query85	1177	273	271	271
query86	457	294	305	294
query87	3320	3049	3106	3049
query88	4277	2468	2438	2438
query89	467	382	375	375
query90	2053	186	188	186
query91	123	99	98	98
query92	61	50	49	49
query93	2025	508	494	494
query94	1128	178	187	178
query95	400	293	290	290
query96	589	278	269	269
query97	3148	2969	2964	2964
query98	291	216	212	212
query99	1173	911	904	904
Total cold run time: 270063 ms
Total hot run time: 186956 ms

morningman
morningman approved these changes May 15, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zy-kkk
Copy link
Member Author

zy-kkk commented May 16, 2024

run performance

@zy-kkk zy-kkk merged commit 1cc2c9a into apache:master May 16, 2024
26 of 28 checks passed
@zy-kkk zy-kkk deleted the split_ck branch May 16, 2024 02:29
dataroaring pushed a commit that referenced this pull request May 16, 2024
@zy-kkk @dataroaring
[refactor](jdbc catalog) split clickhouse jdbc executor (#34794)
14d4fad
zy-kkk added a commit to zy-kkk/doris that referenced this pull request May 22, 2024
@zy-kkk
[refactor](jdbc catalog) split clickhouse jdbc executor (apache#34794)
2854a3f
@zy-kkk zy-kkk mentioned this pull request May 22, 2024
Merged
morningman pushed a commit that referenced this pull request May 22, 2024
@zy-kkk
[refactor](jdbc catalog) split clickhouse jdbc executor (#34794) (#35174
2499038 
)

pick master #34794
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zy-kkk @doris-robot @morningman @CalvinKirs @dataroaring