Skip to content

branch-3.0: [opt](jdbc scan) Add more jdbc scan profile items #46460 #50800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 14, 2025
Merged

branch-3.0: [opt](jdbc scan) Add more jdbc scan profile items #46460 #50800

dataroaring merged 1 commit into from
May 14, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 12, 2025

Cherry-picked from #46460

@zy-kkk
[opt](jdbc scan) Add more jdbc scan profile items (#46460)
d9ed2f0 
### What problem does this PR solve?

Problem Summary:

```
-  GetDataTime:  197.0us (Total time to read data)
    -  CastTime:  2.0us (Convert data from JDBC memory format to Doris BE C++ memory format)
    -  FillBlockTime:  6.0us (Put data into BE Block)
    -  ReadAndFillVectorTableTime:  129.0us (Time to read data and convert to vectorTable)
    -  HasNextTime:  35.0us (The time to obtain each batch of data)
    -  JniSetupTime:  0ns (Start JNI time)
    -  PrepareParamsTime:  24.0us (Preparation parameter time) 
```
@github-actions github-actions bot requested a review from dataroaring as a code owner May 12, 2025 02:58
@hello-stephen
Copy link
Contributor

hello-stephen commented May 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this May 12, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 12, 2025

TPC-H: Total hot run time: 40920 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9ed2f0e58c832c0192a0ccc654f231d9fe62ac2, data reload: false

------ Round 1 ----------------------------------
q1	17726	6802	6631	6631
q2	2070	175	164	164
q3	11625	1096	1230	1096
q4	10500	780	775	775
q5	7748	2877	2917	2877
q6	227	135	134	134
q7	957	619	597	597
q8	9565	2003	2002	2002
q9	6652	6514	6561	6514
q10	7056	2301	2342	2301
q11	457	259	255	255
q12	409	218	213	213
q13	17891	3009	3041	3009
q14	241	204	212	204
q15	520	469	473	469
q16	678	605	586	586
q17	1014	604	544	544
q18	7597	7065	7206	7065
q19	2869	1040	1130	1040
q20	1131	214	210	210
q21	4171	3277	3231	3231
q22	1142	1003	1007	1003
Total cold run time: 112246 ms
Total hot run time: 40920 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6692	6604	6571	6571
q2	332	245	234	234
q3	2941	2893	3035	2893
q4	2053	1862	1880	1862
q5	5771	5837	5812	5812
q6	211	133	133	133
q7	2292	1824	1901	1824
q8	3369	3545	3585	3545
q9	8774	8953	8901	8901
q10	3559	3572	3515	3515
q11	577	500	503	500
q12	822	609	597	597
q13	14209	3179	3146	3146
q14	296	282	274	274
q15	505	453	453	453
q16	704	649	649	649
q17	1848	1645	1616	1616
q18	8318	7760	7763	7760
q19	1673	1505	1498	1498
q20	2096	1882	1834	1834
q21	5529	5478	5364	5364
q22	1152	1031	1058	1031
Total cold run time: 73723 ms
Total hot run time: 60012 ms

@doris-robot
Copy link

doris-robot commented May 12, 2025

TPC-DS: Total hot run time: 197397 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9ed2f0e58c832c0192a0ccc654f231d9fe62ac2, data reload: false

query1	1287	906	903	903
query2	6212	2063	2074	2063
query3	10959	4326	4382	4326
query4	68305	28718	23926	23926
query5	4969	471	467	467
query6	425	185	182	182
query7	5548	313	314	313
query8	299	211	217	211
query9	9051	2575	2577	2575
query10	444	257	264	257
query11	17690	15398	15763	15398
query12	154	105	108	105
query13	1490	446	438	438
query14	10503	7107	6751	6751
query15	210	179	184	179
query16	7166	430	499	430
query17	1063	594	595	594
query18	1903	338	328	328
query19	202	154	161	154
query20	123	116	112	112
query21	204	104	105	104
query22	4599	4443	4492	4443
query23	35523	33843	34038	33843
query24	6200	2933	2966	2933
query25	551	446	442	442
query26	674	180	172	172
query27	1956	353	364	353
query28	4022	2425	2453	2425
query29	711	454	439	439
query30	238	161	169	161
query31	1040	830	834	830
query32	64	56	55	55
query33	484	294	275	275
query34	919	505	522	505
query35	850	735	714	714
query36	1084	966	978	966
query37	118	73	70	70
query38	4006	3953	4018	3953
query39	1530	1477	1458	1458
query40	204	109	98	98
query41	49	49	46	46
query42	119	103	104	103
query43	556	515	509	509
query44	1186	800	817	800
query45	185	167	169	167
query46	1171	726	719	719
query47	2018	1911	1923	1911
query48	484	379	382	379
query49	736	407	398	398
query50	843	433	435	433
query51	7521	7266	7208	7208
query52	99	85	91	85
query53	255	179	186	179
query54	571	462	453	453
query55	77	78	73	73
query56	265	242	248	242
query57	1213	1110	1101	1101
query58	215	200	207	200
query59	3087	2957	2896	2896
query60	282	255	262	255
query61	109	117	105	105
query62	797	712	712	712
query63	226	200	192	192
query64	1468	659	638	638
query65	3274	3199	3237	3199
query66	641	294	303	294
query67	15937	15550	15493	15493
query68	3974	586	563	563
query69	408	262	267	262
query70	1202	1111	1113	1111
query71	326	258	253	253
query72	6192	4118	4021	4021
query73	754	353	363	353
query74	10243	9229	9086	9086
query75	3332	2671	2662	2662
query76	1890	1097	1102	1097
query77	530	271	290	271
query78	10653	9647	9540	9540
query79	1255	601	599	599
query80	849	426	424	424
query81	509	240	236	236
query82	1350	92	87	87
query83	239	141	146	141
query84	281	84	72	72
query85	885	303	299	299
query86	345	293	300	293
query87	4429	4260	4230	4230
query88	3543	2413	2360	2360
query89	417	301	293	293
query90	1894	189	191	189
query91	189	148	149	148
query92	64	47	51	47
query93	1320	552	545	545
query94	742	280	300	280
query95	358	255	247	247
query96	610	294	279	279
query97	3324	3190	3161	3161
query98	214	211	200	200
query99	1545	1307	1314	1307
Total cold run time: 320230 ms
Total hot run time: 197397 ms

@doris-robot
Copy link

doris-robot commented May 12, 2025

ClickBench: Total hot run time: 32.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9ed2f0e58c832c0192a0ccc654f231d9fe62ac2, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.52	0.50	0.52
query6	1.14	0.73	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.10	0.13
query13	0.60	0.60	0.58
query14	2.73	2.76	2.78
query15	0.91	0.83	0.83
query16	0.37	0.38	0.39
query17	1.01	1.03	1.05
query18	0.24	0.22	0.21
query19	1.92	1.80	2.00
query20	0.01	0.00	0.01
query21	15.36	0.60	0.60
query22	2.37	2.65	1.33
query23	17.05	0.93	0.71
query24	3.39	1.65	1.69
query25	0.12	0.13	0.12
query26	0.64	0.14	0.14
query27	0.05	0.04	0.04
query28	8.91	0.56	0.51
query29	12.57	3.32	3.29
query30	0.26	0.07	0.06
query31	2.85	0.40	0.38
query32	3.25	0.47	0.46
query33	3.03	3.02	2.98
query34	16.89	4.53	4.52
query35	4.62	4.54	4.53
query36	0.67	0.48	0.48
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 105.43 s
Total hot run time: 32.42 s

@doris-robot
Copy link

doris-robot commented May 12, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/46) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.96% (10797/26363)
Line Coverage 31.81% (92181/289802)
Region Coverage 30.87% (47563/154052)
Branch Coverage 27.39% (24378/89016)

dataroaring
dataroaring approved these changes May 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit df67c7a into branch-3.0 May 14, 2025
21 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-46460-branch-3.0 branch May 14, 2025 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @dataroaring @zy-kkk