Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(cloud-merge) Fix the coredump because of change_cache_type to origin_type #38518

Merged
merged 2 commits into from
Jul 31, 2024

Conversation

Lchangliang
Copy link
Contributor

When call method change_cache_type_by_mgr, if the cache_type is same as change_type, it will return true directly and execute the next codes and break the old assumptions.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}
fs::create_directories(cache_base_path);
auto sp = SyncPoint::get_instance();
sp->set_call_back("FileBlock::change_cache_type", [](auto&& args) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]

Suggested change
sp->set_call_back("FileBlock::change_cache_type", [](auto&& args) {
h);auto *

}

TEST_F(BlockFileCacheTest, change_cache_type2) {
if (fs::exists(cache_base_path)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]

    ^
Additional context

be/test/io/cache/block_file_cache_test.cpp:4395: 86 lines including whitespace and comments (threshold 80)

    ^

}
fs::create_directories(cache_base_path);
auto sp = SyncPoint::get_instance();
sp->set_call_back("FileBlock::change_cache_type", [](auto&& args) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]

Suggested change
sp->set_call_back("FileBlock::change_cache_type", [](auto&& args) {
h);auto *

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

fs::remove_all(cache_base_path);
}
fs::create_directories(cache_base_path);
auto sp = SyncPoint::get_instance();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]

Suggested change
auto sp = SyncPoint::get_instance();
h);auto *

}
}

TEST_F(BlockFileCacheTest, change_cache_type2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]

    ^
Additional context

be/test/io/cache/block_file_cache_test.cpp:4394: 86 lines including whitespace and comments (threshold 80)

    ^

fs::remove_all(cache_base_path);
}
fs::create_directories(cache_base_path);
auto sp = SyncPoint::get_instance();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]

Suggested change
auto sp = SyncPoint::get_instance();
h);auto *

@doris-robot
Copy link

TPC-H: Total hot run time: 41802 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95046e10ae7aa315cb2beef2f9db8b6b49d10694, data reload: false

------ Round 1 ----------------------------------
q1	20798	4203	4131	4131
q2	2435	208	200	200
q3	11634	1298	1351	1298
q4	10229	875	924	875
q5	7650	2896	2958	2896
q6	220	137	140	137
q7	1044	615	605	605
q8	9431	1942	1940	1940
q9	8514	6584	6594	6584
q10	8698	3824	3807	3807
q11	422	243	257	243
q12	415	230	229	229
q13	17761	2939	2947	2939
q14	271	243	252	243
q15	533	481	488	481
q16	497	412	396	396
q17	967	935	944	935
q18	8061	7281	7229	7229
q19	1382	1231	1203	1203
q20	559	318	334	318
q21	5324	4837	4835	4835
q22	355	278	284	278
Total cold run time: 117200 ms
Total hot run time: 41802 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4047	4022	3989	3989
q2	322	221	216	216
q3	3016	3042	2985	2985
q4	1874	1882	1873	1873
q5	5211	5264	5169	5169
q6	215	131	135	131
q7	2031	1738	1718	1718
q8	3196	3317	3235	3235
q9	8350	8327	8322	8322
q10	3739	3807	3849	3807
q11	544	449	456	449
q12	734	578	528	528
q13	14181	2921	2938	2921
q14	283	273	248	248
q15	514	474	470	470
q16	448	396	399	396
q17	1720	1670	1659	1659
q18	7768	7353	7136	7136
q19	1654	1671	1653	1653
q20	1989	1772	1765	1765
q21	5408	5232	5311	5232
q22	504	462	455	455
Total cold run time: 67748 ms
Total hot run time: 54357 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169618 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95046e10ae7aa315cb2beef2f9db8b6b49d10694, data reload: false

query1	908	368	365	365
query2	6469	1679	1638	1638
query3	6686	218	226	218
query4	19049	17341	17303	17303
query5	4322	521	531	521
query6	292	188	170	170
query7	4620	311	305	305
query8	260	208	212	208
query9	8526	2345	2307	2307
query10	434	297	308	297
query11	10627	9965	9959	9959
query12	142	90	88	88
query13	1630	389	396	389
query14	9782	7773	7681	7681
query15	207	174	166	166
query16	7198	470	444	444
query17	968	575	577	575
query18	1896	336	277	277
query19	192	146	150	146
query20	93	88	86	86
query21	202	106	98	98
query22	4197	4104	3887	3887
query23	33901	32919	32955	32919
query24	10353	3083	3013	3013
query25	672	382	396	382
query26	1805	150	147	147
query27	2953	280	284	280
query28	7019	1954	1964	1954
query29	1303	419	409	409
query30	280	146	148	146
query31	945	768	758	758
query32	103	56	56	56
query33	712	310	314	310
query34	899	478	488	478
query35	863	737	721	721
query36	967	862	842	842
query37	293	80	82	80
query38	2880	2797	2764	2764
query39	879	821	816	816
query40	288	118	113	113
query41	47	45	44	44
query42	116	99	99	99
query43	474	426	416	416
query44	1230	747	736	736
query45	205	178	176	176
query46	1078	810	788	788
query47	1792	1678	1705	1678
query48	403	297	293	293
query49	1193	432	428	428
query50	892	439	449	439
query51	6735	6622	6613	6613
query52	107	89	93	89
query53	265	182	188	182
query54	635	452	468	452
query55	78	76	77	76
query56	289	250	263	250
query57	1118	1053	1068	1053
query58	279	261	266	261
query59	2534	2391	2480	2391
query60	320	268	276	268
query61	104	131	95	95
query62	906	653	670	653
query63	220	193	197	193
query64	5923	1943	1905	1905
query65	3147	3130	3113	3113
query66	1427	351	333	333
query67	15240	14825	14790	14790
query68	4340	581	630	581
query69	437	312	303	303
query70	1134	1085	1071	1071
query71	411	285	280	280
query72	7173	2681	2545	2545
query73	754	333	334	333
query74	6060	5657	5677	5657
query75	3358	2751	2723	2723
query76	2497	1352	1394	1352
query77	466	312	321	312
query78	9468	8827	8854	8827
query79	2324	547	543	543
query80	1124	520	507	507
query81	581	230	230	230
query82	1037	133	134	133
query83	246	174	172	172
query84	267	81	81	81
query85	1365	324	310	310
query86	512	300	299	299
query87	3282	3154	3094	3094
query88	3105	2488	2492	2488
query89	383	286	295	286
query90	1842	257	215	215
query91	132	103	104	103
query92	65	52	51	51
query93	1983	633	639	633
query94	952	300	303	300
query95	386	273	272	272
query96	599	284	288	284
query97	3192	3029	3099	3029
query98	215	197	193	193
query99	1644	1340	1281	1281
Total cold run time: 263906 ms
Total hot run time: 169618 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95046e10ae7aa315cb2beef2f9db8b6b49d10694, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.04	0.04
query4	1.68	0.07	0.08
query5	0.50	0.48	0.48
query6	1.13	0.72	0.70
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.58	0.50	0.51
query10	0.56	0.57	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.61
query14	0.78	0.79	0.80
query15	0.88	0.85	0.86
query16	0.36	0.35	0.35
query17	0.99	1.02	0.96
query18	0.25	0.21	0.22
query19	1.90	1.74	1.79
query20	0.01	0.01	0.01
query21	15.40	0.79	0.68
query22	3.61	6.32	2.05
query23	18.03	1.30	1.25
query24	2.25	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.21	0.22
query27	0.46	0.23	0.23
query28	13.18	1.00	0.97
query29	12.52	3.37	3.31
query30	0.26	0.06	0.06
query31	2.86	0.40	0.40
query32	3.25	0.50	0.48
query33	3.01	2.94	2.95
query34	15.45	4.28	4.25
query35	4.31	4.29	4.30
query36	0.68	0.48	0.47
query37	0.19	0.16	0.15
query38	0.17	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.46 s
Total hot run time: 30.7 s

@Lchangliang
Copy link
Contributor Author

run cloud_p0

@Lchangliang
Copy link
Contributor Author

run cloud_p1

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41733 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8dfbaa2e07266bfee12c7a50d98fc9a8517d787c, data reload: false

------ Round 1 ----------------------------------
q1	18169	4209	4165	4165
q2	2548	213	207	207
q3	11629	1349	1342	1342
q4	11207	842	953	842
q5	9188	2956	2967	2956
q6	223	139	135	135
q7	1035	609	612	609
q8	9426	1843	1938	1843
q9	8489	6615	6616	6615
q10	8696	3827	3844	3827
q11	423	243	243	243
q12	414	230	230	230
q13	17784	2939	2951	2939
q14	275	240	246	240
q15	530	483	496	483
q16	489	402	390	390
q17	947	931	897	897
q18	7913	7269	7226	7226
q19	1386	1228	1213	1213
q20	566	323	335	323
q21	5186	4731	4732	4731
q22	355	284	277	277
Total cold run time: 116878 ms
Total hot run time: 41733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4071	4043	4049	4043
q2	333	224	222	222
q3	2974	3020	2977	2977
q4	1878	1893	1879	1879
q5	5255	5252	5250	5250
q6	217	128	129	128
q7	2078	1687	1715	1687
q8	3227	3287	3257	3257
q9	8302	8295	8258	8258
q10	3773	3858	3801	3801
q11	577	448	451	448
q12	718	587	564	564
q13	16751	2938	2949	2938
q14	291	262	259	259
q15	519	479	481	479
q16	447	398	393	393
q17	1730	1711	1682	1682
q18	7658	7294	7289	7289
q19	1656	1665	1678	1665
q20	1966	1783	1774	1774
q21	5375	5169	5179	5169
q22	546	462	448	448
Total cold run time: 70342 ms
Total hot run time: 54610 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168317 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8dfbaa2e07266bfee12c7a50d98fc9a8517d787c, data reload: false

query1	914	387	373	373
query2	6481	1698	1640	1640
query3	6659	218	228	218
query4	19669	17347	17384	17347
query5	4300	511	525	511
query6	290	173	184	173
query7	4600	313	313	313
query8	277	194	201	194
query9	8510	2306	2310	2306
query10	434	284	263	263
query11	10533	9941	10036	9941
query12	134	87	96	87
query13	1617	382	376	376
query14	9629	8329	6965	6965
query15	207	162	165	162
query16	7108	433	428	428
query17	921	562	543	543
query18	1927	286	279	279
query19	191	147	143	143
query20	96	83	86	83
query21	211	102	98	98
query22	4135	4079	3967	3967
query23	33557	32983	32837	32837
query24	10301	3009	3021	3009
query25	696	387	377	377
query26	1765	150	145	145
query27	2954	277	278	277
query28	7044	1983	1971	1971
query29	1327	411	408	408
query30	280	149	150	149
query31	926	755	750	750
query32	132	53	54	53
query33	701	307	313	307
query34	908	485	472	472
query35	850	735	717	717
query36	981	859	874	859
query37	298	76	74	74
query38	2862	2755	2834	2755
query39	862	800	808	800
query40	277	114	114	114
query41	50	45	43	43
query42	121	97	103	97
query43	471	405	392	392
query44	1202	734	731	731
query45	205	177	179	177
query46	1083	823	781	781
query47	1784	1716	1729	1716
query48	364	299	291	291
query49	1187	414	418	414
query50	912	429	439	429
query51	6728	6638	6660	6638
query52	104	88	89	88
query53	257	186	184	184
query54	642	463	499	463
query55	75	79	75	75
query56	304	256	250	250
query57	1180	1051	1067	1051
query58	278	262	267	262
query59	2678	2372	2283	2283
query60	299	286	264	264
query61	99	96	101	96
query62	918	668	646	646
query63	210	189	176	176
query64	5888	1937	1883	1883
query65	3158	3106	3080	3080
query66	1441	336	333	333
query67	15358	14884	14670	14670
query68	4308	596	598	596
query69	442	301	305	301
query70	1107	1053	1049	1049
query71	417	270	283	270
query72	7056	2648	2484	2484
query73	761	329	326	326
query74	6036	5656	5712	5656
query75	3409	2751	2807	2751
query76	2377	1365	1366	1365
query77	449	312	304	304
query78	9366	8897	8853	8853
query79	1413	536	535	535
query80	1204	513	506	506
query81	554	219	232	219
query82	1079	128	126	126
query83	240	180	169	169
query84	273	83	80	80
query85	1285	312	300	300
query86	374	306	281	281
query87	3243	3169	3074	3074
query88	2963	2481	2478	2478
query89	384	294	279	279
query90	1784	203	188	188
query91	192	103	102	102
query92	61	51	49	49
query93	1456	619	630	619
query94	890	289	290	289
query95	378	263	263	263
query96	599	284	281	281
query97	3216	3027	3057	3027
query98	211	203	200	200
query99	1635	1300	1301	1300
Total cold run time: 261722 ms
Total hot run time: 168317 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8dfbaa2e07266bfee12c7a50d98fc9a8517d787c, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.57	0.50	0.51
query10	0.58	0.57	0.57
query11	0.15	0.12	0.12
query12	0.14	0.13	0.13
query13	0.61	0.60	0.60
query14	0.78	0.79	0.78
query15	0.89	0.86	0.86
query16	0.36	0.36	0.37
query17	0.99	1.01	1.00
query18	0.23	0.22	0.21
query19	1.82	1.74	1.72
query20	0.02	0.01	0.01
query21	15.42	0.78	0.66
query22	3.89	8.34	1.39
query23	17.76	1.37	1.32
query24	2.33	0.22	0.21
query25	0.19	0.08	0.07
query26	0.31	0.21	0.22
query27	0.46	0.23	0.22
query28	13.16	1.00	0.97
query29	12.59	3.34	3.33
query30	0.26	0.06	0.05
query31	2.86	0.40	0.41
query32	3.22	0.49	0.48
query33	2.93	2.98	2.95
query34	15.43	4.28	4.25
query35	4.32	4.31	4.29
query36	0.68	0.48	0.48
query37	0.20	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.11	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.03	0.04
Total cold run time: 107.43 s
Total hot run time: 30.18 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Lchangliang
Copy link
Contributor Author

run p0

@gavinchou gavinchou merged commit 54554fa into apache:master Jul 31, 2024
25 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Aug 2, 2024
…_type (#38518)

When call method `change_cache_type_by_mgr`, if the cache_type is same
as change_type, it will return true directly and execute the next codes
and break the old assumptions.
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Aug 9, 2024
…_type (apache#38518)

When call method `change_cache_type_by_mgr`, if the cache_type is same
as change_type, it will return true directly and execute the next codes
and break the old assumptions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged p0_c reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants