Skip to content

[fix](statistics) External table partition stats improve and add test case. #36884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jibing-Li merged 1 commit into from
Jun 27, 2024
Merged

[fix](statistics) External table partition stats improve and add test case. #36884

Jibing-Li merged 1 commit into from
Jun 27, 2024

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Jun 26, 2024

Add test case for analyze hive external partition table.
Fix a bug of cast HMSExternalTable to OlapTable

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review June 26, 2024 15:53
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-H: Total hot run time: 39522 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed5726e679487da9668b253efe260c10f6c470b5, data reload: false

------ Round 1 ----------------------------------
q1	17623	4311	4280	4280
q2	2012	194	194	194
q3	10461	1218	1123	1123
q4	10196	794	746	746
q5	7463	2619	2607	2607
q6	217	132	131	131
q7	947	591	595	591
q8	9222	2050	2041	2041
q9	8968	6490	6465	6465
q10	8977	3725	3666	3666
q11	443	237	233	233
q12	476	229	225	225
q13	17759	2979	3010	2979
q14	261	236	219	219
q15	537	478	490	478
q16	529	373	366	366
q17	948	605	732	605
q18	8024	7420	7358	7358
q19	6505	1503	1421	1421
q20	651	309	317	309
q21	4960	3149	3283	3149
q22	392	340	336	336
Total cold run time: 117571 ms
Total hot run time: 39522 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4249	4198	4198
q2	366	264	264	264
q3	2991	2816	2801	2801
q4	1929	1751	1673	1673
q5	5632	5552	5465	5465
q6	218	125	126	125
q7	2183	1873	1870	1870
q8	3227	3410	3404	3404
q9	8645	8567	8801	8567
q10	4100	3821	3648	3648
q11	597	516	498	498
q12	820	676	631	631
q13	16068	3209	3163	3163
q14	314	285	265	265
q15	549	480	507	480
q16	496	428	419	419
q17	1796	1534	1478	1478
q18	7960	7808	7787	7787
q19	1845	1681	1579	1579
q20	3071	1900	1914	1900
q21	5109	5039	4924	4924
q22	815	556	570	556
Total cold run time: 73103 ms
Total hot run time: 55695 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-DS: Total hot run time: 174501 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed5726e679487da9668b253efe260c10f6c470b5, data reload: false

query1	927	387	367	367
query2	6466	2590	2261	2261
query3	6631	201	207	201
query4	19799	17493	17276	17276
query5	3619	457	479	457
query6	277	168	164	164
query7	4591	299	293	293
query8	319	295	285	285
query9	8477	2461	2450	2450
query10	570	296	284	284
query11	10688	10071	10082	10071
query12	117	83	84	83
query13	1644	378	379	378
query14	10111	7580	7815	7580
query15	235	188	182	182
query16	7626	267	260	260
query17	1928	566	505	505
query18	1631	269	266	266
query19	205	145	147	145
query20	87	81	84	81
query21	209	133	123	123
query22	4411	4239	4030	4030
query23	33958	33776	33675	33675
query24	10928	2942	2917	2917
query25	624	382	372	372
query26	1135	153	152	152
query27	2381	333	320	320
query28	7009	2154	2163	2154
query29	878	613	630	613
query30	226	152	151	151
query31	970	764	764	764
query32	92	53	55	53
query33	759	285	288	285
query34	1053	478	488	478
query35	728	654	609	609
query36	1116	986	958	958
query37	150	77	79	77
query38	2979	2841	2859	2841
query39	926	853	834	834
query40	202	129	125	125
query41	55	52	52	52
query42	113	99	98	98
query43	591	569	551	551
query44	1205	728	721	721
query45	206	165	170	165
query46	1069	697	687	687
query47	1875	1770	1762	1762
query48	360	302	303	302
query49	835	394	402	394
query50	767	402	377	377
query51	6831	6741	6824	6741
query52	98	93	93	93
query53	361	286	286	286
query54	871	456	444	444
query55	73	72	72	72
query56	281	259	296	259
query57	1100	1046	1063	1046
query58	249	239	232	232
query59	3272	3252	3165	3165
query60	290	267	268	267
query61	91	91	91	91
query62	620	450	438	438
query63	319	288	283	283
query64	8882	2221	1723	1723
query65	3136	3093	3109	3093
query66	757	337	343	337
query67	15503	15236	14965	14965
query68	4579	525	543	525
query69	519	383	346	346
query70	1195	1159	1116	1116
query71	423	282	275	275
query72	6937	5349	5835	5349
query73	751	326	326	326
query74	5909	5588	5484	5484
query75	3333	2664	2698	2664
query76	2408	929	917	917
query77	472	299	291	291
query78	10409	9865	9837	9837
query79	2626	504	506	504
query80	2166	453	449	449
query81	552	218	217	217
query82	1397	111	102	102
query83	321	172	170	170
query84	273	97	83	83
query85	1359	278	265	265
query86	468	313	336	313
query87	3309	3074	3081	3074
query88	4109	2447	2420	2420
query89	469	377	366	366
query90	1754	181	187	181
query91	133	99	99	99
query92	55	46	48	46
query93	2568	490	501	490
query94	1075	186	187	186
query95	458	310	319	310
query96	591	274	271	271
query97	3248	3048	3031	3031
query98	224	202	192	192
query99	1333	843	838	838
Total cold run time: 271915 ms
Total hot run time: 174501 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed5726e679487da9668b253efe260c10f6c470b5, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.05	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.08
query5	0.54	0.49	0.49
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.56	0.48	0.49
query10	0.55	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.60
query14	0.75	0.77	0.81
query15	0.84	0.81	0.81
query16	0.37	0.36	0.36
query17	1.02	1.00	1.00
query18	0.23	0.23	0.23
query19	1.74	1.83	1.71
query20	0.02	0.01	0.00
query21	15.44	0.74	0.65
query22	4.56	6.95	2.14
query23	18.32	1.36	1.24
query24	1.98	0.23	0.21
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.31	1.00	0.98
query29	12.61	3.24	3.21
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.28	0.47	0.47
query33	2.89	2.92	2.90
query34	17.11	4.45	4.40
query35	4.48	4.52	4.44
query36	0.66	0.49	0.47
query37	0.19	0.14	0.14
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 30.66 s

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 27, 2024

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by anyone and no changes requested.

morningman
morningman approved these changes Jun 27, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jibing-Li Jibing-Li merged commit cc48413 into apache:master Jun 27, 2024
@Jibing-Li Jibing-Li deleted the external branch June 27, 2024 08:17
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@Jibing-Li @dataroaring
[fix](statistics) External table partition stats improve and add test…
81aa4c3 
… case. (#36884)

Add test case for analyze hive external partition table.
Fix a bug of cast HMSExternalTable to OlapTable
@cambyzju cambyzju mentioned this pull request Aug 6, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 6, 2024
@cambyzju
[opt](log) Remove unnecessary log for analysis (#38943)
2540835 
In master branch, we already fixed in this pr:
#36884

Here cherry-pick to branch-2.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged not-merge/2.1 reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Jibing-Li @doris-robot @morningman @morrySnow @dataroaring