Skip to content

branch-3.0: [improvement](statistics)Async drop table stats while doing truncate and schema change. #45923#45997

Merged
Jibing-Li merged 1 commit intobranch-3.0from
auto-pick-45923-branch-3.0
Dec 26, 2024
Merged

branch-3.0: [improvement](statistics)Async drop table stats while doing truncate and schema change. #45923#45997
Jibing-Li merged 1 commit intobranch-3.0from
auto-pick-45923-branch-3.0

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #45923

…and schema change. (#45923)

### What problem does this PR solve?

Async drop table stats while doing truncate and schema change. Truncate
can schema change operation may hold table's write lock. And these two
operations will trigger drop old stats info. Drop stats with write lock
holding may bring performance issue.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
@Thearas
Copy link
Contributor

Thearas commented Dec 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 26, 2024
@Thearas
Copy link
Contributor

Thearas commented Dec 26, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41081 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 706d1ff8b11e4ecc7b95e04159524adcdeeb08c9, data reload: false

------ Round 1 ----------------------------------
q1	17584	7504	7342	7342
q2	2070	172	165	165
q3	10622	1167	1200	1167
q4	10242	712	764	712
q5	7730	2873	2841	2841
q6	233	144	145	144
q7	961	630	605	605
q8	9351	1988	2023	1988
q9	6599	6394	6401	6394
q10	6984	2306	2341	2306
q11	473	273	260	260
q12	393	214	209	209
q13	17783	2986	3003	2986
q14	248	214	219	214
q15	567	535	528	528
q16	697	621	610	610
q17	975	573	569	569
q18	7319	6598	6758	6598
q19	1386	1061	1040	1040
q20	478	204	206	204
q21	4003	3336	3222	3222
q22	1070	993	977	977
Total cold run time: 107768 ms
Total hot run time: 41081 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7262	7216	7213	7213
q2	341	236	224	224
q3	2888	2820	2763	2763
q4	1953	1725	1679	1679
q5	5407	5450	5441	5441
q6	217	135	132	132
q7	2166	1697	1689	1689
q8	3243	3422	3424	3422
q9	8593	8610	8646	8610
q10	3528	3456	3414	3414
q11	585	501	500	500
q12	802	577	591	577
q13	7814	3008	3018	3008
q14	286	275	261	261
q15	574	518	511	511
q16	725	684	660	660
q17	1814	1568	1566	1566
q18	7865	7552	7417	7417
q19	1641	1614	1545	1545
q20	2021	1816	1847	1816
q21	5426	5163	5211	5163
q22	1098	999	996	996
Total cold run time: 66249 ms
Total hot run time: 58607 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 706d1ff8b11e4ecc7b95e04159524adcdeeb08c9, data reload: false

query1	979	370	362	362
query2	6516	2127	2094	2094
query3	6709	219	226	219
query4	33758	23534	23659	23534
query5	4402	455	452	452
query6	269	168	180	168
query7	4640	321	323	321
query8	290	239	234	234
query9	9540	2679	2676	2676
query10	480	271	264	264
query11	18285	15143	15422	15143
query12	151	100	99	99
query13	1628	452	431	431
query14	9633	7408	7500	7408
query15	256	172	179	172
query16	8072	455	439	439
query17	1690	593	564	564
query18	2163	315	317	315
query19	381	163	162	162
query20	123	107	110	107
query21	65	48	47	47
query22	4456	4249	4238	4238
query23	34906	34311	34180	34180
query24	11178	2960	2885	2885
query25	702	409	412	409
query26	1491	174	175	174
query27	2846	307	301	301
query28	8071	2461	2437	2437
query29	945	454	441	441
query30	339	167	167	167
query31	1025	814	800	800
query32	92	56	61	56
query33	795	307	295	295
query34	933	508	522	508
query35	862	730	723	723
query36	1083	942	963	942
query37	146	81	78	78
query38	4121	3892	3925	3892
query39	1499	1424	1432	1424
query40	224	86	85	85
query41	56	50	55	50
query42	118	102	103	102
query43	552	507	514	507
query44	1261	826	815	815
query45	189	171	169	169
query46	1146	708	721	708
query47	1940	1838	1886	1838
query48	480	378	388	378
query49	1170	407	409	407
query50	822	421	425	421
query51	7289	7079	7043	7043
query52	104	90	92	90
query53	261	185	184	184
query54	1197	466	470	466
query55	80	82	80	80
query56	307	266	266	266
query57	1239	1130	1100	1100
query58	249	218	213	213
query59	3328	3037	3009	3009
query60	278	249	254	249
query61	105	109	106	106
query62	885	661	671	661
query63	219	199	187	187
query64	5435	655	619	619
query65	3332	3189	3205	3189
query66	1440	322	336	322
query67	16234	15661	15470	15470
query68	4897	565	574	565
query69	439	299	261	261
query70	1230	1124	1083	1083
query71	408	263	259	259
query72	6579	4042	3994	3994
query73	774	344	358	344
query74	10369	9036	8993	8993
query75	3425	2611	2635	2611
query76	2947	1061	1082	1061
query77	460	286	274	274
query78	10374	9609	9543	9543
query79	1758	609	610	609
query80	1042	446	433	433
query81	554	240	239	239
query82	938	122	117	117
query83	223	150	154	150
query84	235	81	79	79
query85	1246	303	293	293
query86	362	319	301	301
query87	4442	4378	4325	4325
query88	3676	2400	2358	2358
query89	406	300	293	293
query90	1936	189	189	189
query91	186	148	149	148
query92	58	52	52	52
query93	1101	549	543	543
query94	905	301	300	300
query95	357	258	255	255
query96	608	282	281	281
query97	3294	3185	3235	3185
query98	211	204	207	204
query99	1552	1319	1317	1317
Total cold run time: 302452 ms
Total hot run time: 192554 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 706d1ff8b11e4ecc7b95e04159524adcdeeb08c9, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.02
query3	0.24	0.06	0.06
query4	1.63	0.09	0.10
query5	0.52	0.49	0.52
query6	1.15	0.74	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.54	0.59
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.62	0.60	0.59
query14	2.91	3.01	2.93
query15	0.90	0.82	0.82
query16	0.38	0.37	0.37
query17	1.03	0.99	1.03
query18	0.23	0.23	0.21
query19	1.97	1.89	2.04
query20	0.01	0.01	0.02
query21	15.36	0.62	0.58
query22	2.48	2.10	2.62
query23	16.94	0.98	0.87
query24	3.10	1.26	1.83
query25	0.22	0.22	0.17
query26	0.45	0.14	0.15
query27	0.04	0.04	0.03
query28	9.86	1.10	1.06
query29	12.54	3.28	3.31
query30	0.24	0.06	0.05
query31	2.87	0.40	0.38
query32	3.23	0.45	0.46
query33	3.00	2.98	3.07
query34	17.39	4.46	4.50
query35	4.51	4.52	4.48
query36	0.67	0.49	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.54 s
Total hot run time: 33.59 s

@Jibing-Li
Copy link
Contributor

run p0

1 similar comment
@Jibing-Li
Copy link
Contributor

run p0

@Jibing-Li Jibing-Li merged commit ee733c0 into branch-3.0 Dec 26, 2024
@github-actions github-actions bot deleted the auto-pick-45923-branch-3.0 branch December 26, 2024 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants