Skip to content

[fix](cloud) fix auto analyze triggering failure#32139

Merged
dataroaring merged 3 commits intoapache:masterfrom
freemandealer:fix-auto-analyze
Mar 12, 2024
Merged

[fix](cloud) fix auto analyze triggering failure#32139
dataroaring merged 3 commits intoapache:masterfrom
freemandealer:fix-auto-analyze

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Mar 12, 2024

StatisticsCollector only collect when replicas are normal. In cloud mode, this check needs getCloudCluster before execution. Otherwise, check will be a dead-end failure.

image

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

StatisticsCollector only collect when replicas are normal. In cloud
mode, this check needs setCloudCluster before execution. Otherwise,
check will be a dead-end failure.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Copy link
Contributor

@gavinchou gavinchou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

Comment on lines +454 to +464
try (AutoCloseConnectContext r = buildConnectContext()) {
r.connectContext.setCloudCluster();
for (OlapTable table : statsTbls) {
for (Partition partition : table.getPartitions()) {
if (partition.getBaseIndex().getTablets().stream()
.anyMatch(t -> t.getNormalReplicaBackendIds().isEmpty())) {
return false;
}
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
try (AutoCloseConnectContext r = buildConnectContext()) {
r.connectContext.setCloudCluster();
for (OlapTable table : statsTbls) {
for (Partition partition : table.getPartitions()) {
if (partition.getBaseIndex().getTablets().stream()
.anyMatch(t -> t.getNormalReplicaBackendIds().isEmpty())) {
return false;
}
}
}
}
r.connectContext.getCloudCluster();

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
Copy link
Contributor Author

run buildall

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 36964 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 00f2b4ba6bd0100215870042666acb7297b6e3ee, data reload: false

------ Round 1 ----------------------------------
q1	17650	4232	4143	4143
q2	2015	151	143	143
q3	10571	903	890	890
q4	4654	685	697	685
q5	7616	2779	2829	2779
q6	184	151	123	123
q7	1182	814	812	812
q8	9477	2020	1995	1995
q9	7534	6530	6449	6449
q10	8228	2492	2552	2492
q11	432	215	219	215
q12	779	296	286	286
q13	19453	3201	3213	3201
q14	288	273	290	273
q15	508	513	482	482
q16	890	425	448	425
q17	1067	577	573	573
q18	7356	5812	5885	5812
q19	1567	1424	1464	1424
q20	551	307	283	283
q21	7471	3187	3195	3187
q22	804	292	315	292
Total cold run time: 110277 ms
Total hot run time: 36964 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4086	4019	4065	4019
q2	316	228	219	219
q3	3023	2828	2823	2823
q4	1776	1552	1526	1526
q5	5159	5214	5215	5214
q6	203	114	114	114
q7	2210	1837	1853	1837
q8	3146	3277	3243	3243
q9	8525	8519	8523	8519
q10	6216	3707	3647	3647
q11	531	440	433	433
q12	729	553	575	553
q13	11996	2857	2856	2856
q14	272	258	244	244
q15	480	440	442	440
q16	465	410	403	403
q17	1733	1475	1454	1454
q18	7670	7359	7043	7043
q19	6123	1513	1553	1513
q20	1909	1707	1722	1707
q21	4739	4551	4589	4551
q22	523	443	475	443
Total cold run time: 71830 ms
Total hot run time: 52801 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187423 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 00f2b4ba6bd0100215870042666acb7297b6e3ee, data reload: false

query1	925	359	350	350
query2	7425	1959	2025	1959
query3	6704	205	212	205
query4	31150	20696	20688	20688
query5	4299	470	397	397
query6	270	177	166	166
query7	4634	290	293	290
query8	226	182	168	168
query9	9932	2369	2322	2322
query10	457	234	231	231
query11	17289	14350	14285	14285
query12	134	86	83	83
query13	1628	428	453	428
query14	12318	10521	10603	10521
query15	256	210	199	199
query16	8071	267	259	259
query17	1926	568	537	537
query18	2080	295	284	284
query19	257	160	153	153
query20	91	85	81	81
query21	200	135	122	122
query22	4654	4468	4398	4398
query23	31712	31299	31096	31096
query24	10776	2897	2829	2829
query25	617	384	380	380
query26	1229	152	155	152
query27	2614	365	353	353
query28	7712	1942	1922	1922
query29	878	650	618	618
query30	308	148	146	146
query31	974	722	719	719
query32	97	63	59	59
query33	769	259	254	254
query34	1080	475	477	475
query35	826	601	628	601
query36	976	850	851	850
query37	112	75	76	75
query38	3301	3148	3122	3122
query39	1440	1410	1388	1388
query40	210	110	110	110
query41	51	50	48	48
query42	101	101	96	96
query43	485	460	456	456
query44	1075	720	709	709
query45	263	256	263	256
query46	1092	701	681	681
query47	1668	1596	1604	1596
query48	446	345	361	345
query49	1134	335	345	335
query50	752	385	382	382
query51	6795	6667	6591	6591
query52	109	93	85	85
query53	339	279	276	276
query54	323	240	235	235
query55	87	84	77	77
query56	254	230	259	230
query57	1068	1032	1027	1027
query58	234	201	208	201
query59	2666	2529	2619	2529
query60	262	253	257	253
query61	116	114	112	112
query62	589	397	398	397
query63	300	280	282	280
query64	5596	3772	3582	3582
query65	3057	3012	3004	3004
query66	835	319	326	319
query67	14818	15543	14683	14683
query68	8716	553	536	536
query69	628	373	384	373
query70	1189	1104	1175	1104
query71	552	272	285	272
query72	6399	2663	2490	2490
query73	724	317	327	317
query74	7900	6614	6589	6589
query75	9664	8089	8023	8023
query76	5171	979	937	937
query77	679	277	260	260
query78	10365	9644	9608	9608
query79	9864	515	523	515
query80	1671	373	455	373
query81	535	215	210	210
query82	885	208	198	198
query83	219	144	147	144
query84	284	80	76	76
query85	1448	328	318	318
query86	476	304	306	304
query87	3465	3205	3192	3192
query88	5035	2354	2337	2337
query89	513	368	370	368
query90	2000	174	174	174
query91	157	125	128	125
query92	66	47	48	47
query93	6873	516	490	490
query94	1092	176	171	171
query95	422	325	329	325
query96	622	268	266	266
query97	4419	4282	4259	4259
query98	229	222	220	220
query99	1120	742	748	742
Total cold run time: 318454 ms
Total hot run time: 187423 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 00f2b4ba6bd0100215870042666acb7297b6e3ee, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.02
query3	0.23	0.06	0.06
query4	1.65	0.09	0.09
query5	0.51	0.47	0.51
query6	1.18	0.61	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.49	0.49
query10	0.55	0.54	0.54
query11	0.13	0.09	0.09
query12	0.11	0.10	0.09
query13	0.57	0.57	0.56
query14	0.73	0.76	0.76
query15	0.80	0.77	0.78
query16	0.40	0.40	0.38
query17	1.03	1.03	1.04
query18	0.19	0.26	0.22
query19	1.81	1.68	1.68
query20	0.01	0.01	0.01
query21	15.40	0.54	0.55
query22	2.43	1.89	1.49
query23	17.32	0.81	0.94
query24	3.17	0.72	0.99
query25	0.17	0.13	0.03
query26	0.58	0.12	0.13
query27	0.03	0.02	0.02
query28	11.32	0.85	0.85
query29	12.53	3.23	3.23
query30	0.63	0.58	0.53
query31	2.80	0.33	0.34
query32	3.36	0.45	0.43
query33	2.80	2.84	2.78
query34	16.24	4.38	4.26
query35	4.31	4.32	4.31
query36	1.06	1.01	1.00
query37	0.07	0.05	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.20	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.01	0.02
query43	0.02	0.03	0.02
Total cold run time: 105.24 s
Total hot run time: 29.67 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 00f2b4ba6bd0100215870042666acb7297b6e3ee with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       17.5 seconds inserted 10000000 Rows, about 571K ops/s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit dc7fcd0 into apache:master Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants