Skip to content

Conversation

@Jibing-Li
Copy link
Contributor

The older Doris version doesn't support partition stats collection, so when user upgrade their cluster, the metadata doesn't contain partition level update rows for each table. In this case, auto collector thought this table haven't collect partition stats, which will trigger a new collection. Since the partition level update rows are only updated after load operation, if user stop loading, auto analyzing will keep collecting this table for ever.
This pr is to fix this problem, set the partition update rows to 0 for the old version tables.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review June 5, 2024 09:02
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2024

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40469 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false

------ Round 1 ----------------------------------
q1	17930	4681	4344	4344
q2	2990	194	203	194
q3	11793	1151	1153	1151
q4	10430	768	754	754
q5	7939	2662	2618	2618
q6	226	142	135	135
q7	968	599	586	586
q8	9225	2051	2099	2051
q9	8826	6485	6449	6449
q10	8823	3723	3700	3700
q11	438	240	234	234
q12	446	233	227	227
q13	17762	3015	2942	2942
q14	275	215	227	215
q15	503	463	464	463
q16	531	379	375	375
q17	968	669	684	669
q18	8033	7461	7423	7423
q19	7623	1423	1513	1423
q20	629	306	306	306
q21	5026	3877	3979	3877
q22	391	339	333	333
Total cold run time: 121775 ms
Total hot run time: 40469 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4248	4198	4198
q2	376	261	271	261
q3	2966	2759	2785	2759
q4	1857	1646	1680	1646
q5	5245	5309	5276	5276
q6	216	124	125	124
q7	2088	1779	1756	1756
q8	3214	3283	3339	3283
q9	8272	8346	8341	8341
q10	3872	3695	3671	3671
q11	581	497	489	489
q12	779	586	634	586
q13	17232	2938	2988	2938
q14	303	258	286	258
q15	505	477	478	477
q16	505	410	410	410
q17	1777	1480	1493	1480
q18	7517	7461	7419	7419
q19	1638	1600	1580	1580
q20	1949	1814	1808	1808
q21	4799	4664	4807	4664
q22	627	563	523	523
Total cold run time: 70705 ms
Total hot run time: 53947 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170853 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false

query1	929	373	361	361
query2	6466	2467	2280	2280
query3	6653	205	205	205
query4	19205	17349	17392	17349
query5	4186	450	458	450
query6	258	160	169	160
query7	4588	300	289	289
query8	336	277	273	273
query9	8471	2457	2426	2426
query10	440	299	293	293
query11	10644	10122	10066	10066
query12	137	90	86	86
query13	1628	367	364	364
query14	10202	7625	7747	7625
query15	224	190	188	188
query16	7814	288	257	257
query17	1775	512	500	500
query18	1958	275	271	271
query19	202	152	158	152
query20	92	82	85	82
query21	206	133	134	133
query22	4287	4148	4022	4022
query23	33637	33197	33109	33109
query24	12022	2879	2805	2805
query25	676	351	357	351
query26	1812	154	151	151
query27	2984	312	317	312
query28	7709	2063	2068	2063
query29	1137	641	608	608
query30	281	146	149	146
query31	971	746	740	740
query32	97	51	53	51
query33	762	288	282	282
query34	958	474	476	474
query35	738	624	618	618
query36	1118	961	907	907
query37	180	72	73	72
query38	2872	2779	2747	2747
query39	856	801	810	801
query40	276	127	126	126
query41	59	50	51	50
query42	121	99	102	99
query43	557	567	540	540
query44	1243	737	752	737
query45	196	174	164	164
query46	1082	717	725	717
query47	1817	1759	1847	1759
query48	371	292	298	292
query49	1176	412	413	412
query50	789	385	389	385
query51	6778	6700	6606	6606
query52	102	95	95	95
query53	357	291	289	289
query54	986	463	438	438
query55	75	71	74	71
query56	284	266	260	260
query57	1146	1044	1044	1044
query58	255	286	253	253
query59	3535	3294	3338	3294
query60	293	301	267	267
query61	86	89	84	84
query62	628	442	440	440
query63	326	290	286	286
query64	9751	2196	1826	1826
query65	3204	3130	3151	3130
query66	1374	328	338	328
query67	15444	15040	15024	15024
query68	4576	531	526	526
query69	451	301	295	295
query70	1094	1100	1158	1100
query71	415	286	296	286
query72	7055	5860	2707	2707
query73	768	328	319	319
query74	5929	5504	5471	5471
query75	3407	2641	2685	2641
query76	2592	930	903	903
query77	438	297	313	297
query78	10359	9685	9770	9685
query79	2267	514	511	511
query80	848	467	460	460
query81	565	221	222	221
query82	762	104	103	103
query83	279	167	172	167
query84	253	85	88	85
query85	1629	267	260	260
query86	504	282	310	282
query87	3274	3132	3082	3082
query88	3972	2357	2347	2347
query89	485	393	378	378
query90	1821	194	189	189
query91	125	93	102	93
query92	60	50	52	50
query93	2138	514	506	506
query94	1274	192	192	192
query95	408	310	319	310
query96	587	269	269	269
query97	3200	3056	2987	2987
query98	237	221	212	212
query99	1159	834	837	834
Total cold run time: 274753 ms
Total hot run time: 170853 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.10	0.10
query5	0.51	0.50	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.46	0.48
query10	0.53	0.54	0.53
query11	0.15	0.10	0.11
query12	0.15	0.11	0.12
query13	0.59	0.58	0.60
query14	0.79	0.76	0.78
query15	0.83	0.81	0.81
query16	0.38	0.35	0.36
query17	1.01	0.99	1.00
query18	0.22	0.24	0.27
query19	1.81	1.80	1.81
query20	0.02	0.01	0.01
query21	15.47	0.74	0.68
query22	4.94	6.89	1.52
query23	18.34	1.38	1.21
query24	1.86	0.20	0.20
query25	0.13	0.09	0.09
query26	0.27	0.17	0.18
query27	0.09	0.08	0.09
query28	13.40	1.03	1.01
query29	12.69	3.31	3.28
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.47	0.47
query33	2.92	2.90	2.90
query34	17.01	4.38	4.40
query35	4.52	4.46	4.46
query36	0.65	0.46	0.48
query37	0.17	0.15	0.15
query38	0.15	0.13	0.13
query39	0.04	0.03	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.15 s
Total hot run time: 30.14 s

@Jibing-Li Jibing-Li merged commit b009cc0 into apache:master Jun 5, 2024
@Jibing-Li Jibing-Li deleted the fix branch June 6, 2024 05:04
dataroaring pushed a commit that referenced this pull request Jun 7, 2024
)

The older Doris version doesn't support partition stats collection, so
when user upgrade their cluster, the metadata doesn't contain partition
level update rows for each table. In this case, auto collector thought
this table haven't collect partition stats, which will trigger a new
collection. Since the partition level update rows are only updated after
load operation, if user stop loading, auto analyzing will keep
collecting this table for ever.
This pr is to fix this problem, set the partition update rows to 0 for
the old version tables.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged not-merge/2.1 reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants