Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](schema scan) Use async scanner for schema scanners #38403

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[Improvement](schema scan) Use async scanner for schema scanners #38403

wants to merge 5 commits into from
+264 −145

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Jul 26, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/exec/schema_scanner/schema_active_queries_scanner.cpp
}

Status SchemaActiveQueriesScanner::_get_active_queries_block_from_fe() {
void SchemaActiveQueriesScanner::_get_next_block_internal() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_get_next_block_internal' exceeds recommended size/complexity thresholds [readability-function-size]

void SchemaActiveQueriesScanner::_get_next_block_internal() {
                                 ^
Additional context

be/src/exec/schema_scanner/schema_active_queries_scanner.cpp:53: 97 lines including whitespace and comments (threshold 80)

void SchemaActiveQueriesScanner::_get_next_block_internal() {
                                 ^

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39114 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2e98b509bb7ddbbcab7caa048081b718e5aa134, data reload: false

------ Round 1 ----------------------------------
q1	17627	4458	4278	4278
q2	2006	200	215	200
q3	10421	1217	1048	1048
q4	10140	720	673	673
q5	7583	2723	2680	2680
q6	224	137	141	137
q7	965	595	587	587
q8	9217	1905	1929	1905
q9	8904	6629	6557	6557
q10	8821	3775	3844	3775
q11	447	251	252	251
q12	490	231	222	222
q13	18387	2996	2963	2963
q14	269	231	229	229
q15	531	488	479	479
q16	484	389	374	374
q17	981	646	744	646
q18	8165	7433	7328	7328
q19	4707	932	1008	932
q20	687	340	326	326
q21	4968	3231	3277	3231
q22	365	297	293	293
Total cold run time: 116389 ms
Total hot run time: 39114 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4414	4281	4283	4281
q2	371	265	265	265
q3	2989	2810	2898	2810
q4	1969	1702	1682	1682
q5	5623	5590	5523	5523
q6	229	132	139	132
q7	2173	1798	1839	1798
q8	3308	3431	3402	3402
q9	8798	8770	8796	8770
q10	4128	3865	3851	3851
q11	607	485	489	485
q12	757	614	608	608
q13	15865	3192	3191	3191
q14	323	282	281	281
q15	552	483	500	483
q16	480	440	426	426
q17	1814	1560	1504	1504
q18	8128	8152	7939	7939
q19	3093	1636	1655	1636
q20	2181	1939	1874	1874
q21	8445	4947	4826	4826
q22	601	527	512	512
Total cold run time: 76848 ms
Total hot run time: 56279 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2e98b509bb7ddbbcab7caa048081b718e5aa134, data reload: false

query1	911	384	389	384
query2	6438	1925	1838	1838
query3	6654	203	213	203
query4	28500	17481	17161	17161
query5	3665	480	497	480
query6	276	177	174	174
query7	4575	288	289	288
query8	269	204	190	190
query9	8458	2458	2448	2448
query10	448	283	270	270
query11	12081	10008	10246	10008
query12	117	83	82	82
query13	1622	353	360	353
query14	10478	7622	6717	6717
query15	222	162	175	162
query16	7583	476	484	476
query17	1306	546	527	527
query18	1935	279	274	274
query19	183	137	140	137
query20	91	81	82	81
query21	198	99	96	96
query22	4148	3994	3926	3926
query23	34009	33991	33597	33597
query24	10557	2963	2936	2936
query25	604	397	425	397
query26	706	149	148	148
query27	2272	280	274	274
query28	6116	2079	2055	2055
query29	792	430	415	415
query30	259	157	159	157
query31	985	731	771	731
query32	93	56	59	56
query33	688	334	319	319
query34	872	502	489	489
query35	863	762	770	762
query36	1118	970	960	960
query37	150	88	98	88
query38	2998	2855	2856	2855
query39	865	807	815	807
query40	193	114	109	109
query41	44	46	43	43
query42	120	95	94	94
query43	525	473	458	458
query44	1102	721	715	715
query45	208	177	171	171
query46	1071	729	756	729
query47	1828	1735	1765	1735
query48	384	298	293	293
query49	846	409	412	409
query50	790	398	396	396
query51	6776	6609	6698	6609
query52	96	90	84	84
query53	255	177	187	177
query54	842	445	440	440
query55	75	74	71	71
query56	295	286	305	286
query57	1117	1039	1050	1039
query58	278	274	295	274
query59	2835	2628	2689	2628
query60	307	297	289	289
query61	96	97	93	93
query62	774	666	615	615
query63	210	174	175	174
query64	9147	2276	1692	1692
query65	3191	3089	3080	3080
query66	757	342	346	342
query67	15223	14701	15052	14701
query68	4897	548	536	536
query69	578	400	333	333
query70	1136	1026	1145	1026
query71	438	298	287	287
query72	7709	5594	5786	5594
query73	767	329	328	328
query74	6259	5666	5610	5610
query75	3441	2665	2664	2664
query76	3044	908	966	908
query77	605	318	318	318
query78	9734	8977	8920	8920
query79	1597	533	530	530
query80	2348	493	488	488
query81	641	216	217	216
query82	824	139	131	131
query83	311	167	169	167
query84	270	78	78	78
query85	1548	310	303	303
query86	483	307	302	302
query87	3195	3071	3096	3071
query88	4214	2466	2456	2456
query89	399	288	287	287
query90	1964	191	199	191
query91	129	101	101	101
query92	64	47	52	47
query93	2179	523	516	516
query94	1045	281	312	281
query95	352	262	265	262
query96	590	277	277	277
query97	3147	3038	3017	3017
query98	223	202	204	202
query99	1671	1222	1265	1222
Total cold run time: 279292 ms
Total hot run time: 171736 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2e98b509bb7ddbbcab7caa048081b718e5aa134, data reload: false

query1	0.05	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.04	0.04
query4	1.66	0.08	0.07
query5	0.50	0.50	0.48
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.56	0.48	0.48
query10	0.54	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.58	0.58
query14	0.76	0.81	0.77
query15	0.84	0.80	0.83
query16	0.37	0.37	0.37
query17	0.98	1.01	0.98
query18	0.22	0.21	0.22
query19	1.81	1.68	1.74
query20	0.01	0.00	0.00
query21	15.40	0.77	0.66
query22	3.99	6.56	2.68
query23	18.41	1.33	1.22
query24	2.17	0.23	0.22
query25	0.16	0.08	0.09
query26	0.31	0.21	0.20
query27	0.46	0.23	0.23
query28	13.21	1.02	1.00
query29	12.61	3.34	3.29
query30	0.25	0.05	0.06
query31	2.87	0.39	0.38
query32	3.28	0.48	0.48
query33	2.88	2.96	2.92
query34	16.92	4.35	4.39
query35	4.43	4.43	4.42
query36	0.66	0.48	0.46
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.45 s
Total hot run time: 31.28 s

github-actions[bot]
github-actions bot reviewed Jul 26, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/exec/schema_scanner/schema_active_queries_scanner.cpp
}

Status SchemaActiveQueriesScanner::_get_active_queries_block_from_fe() {
void SchemaActiveQueriesScanner::_get_next_block_internal() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_get_next_block_internal' exceeds recommended size/complexity thresholds [readability-function-size]

void SchemaActiveQueriesScanner::_get_next_block_internal() {
                                 ^
Additional context

be/src/exec/schema_scanner/schema_active_queries_scanner.cpp:53: 98 lines including whitespace and comments (threshold 80)

void SchemaActiveQueriesScanner::_get_next_block_internal() {
                                 ^

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39404 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 762e81cafb9c10349402fce13acbabe09b1438b0, data reload: false

------ Round 1 ----------------------------------
q1	18031	4690	4429	4429
q2	2577	207	203	203
q3	11124	1303	1071	1071
q4	11230	768	704	704
q5	8322	2778	2727	2727
q6	226	139	136	136
q7	976	606	590	590
q8	9229	1922	1909	1909
q9	8661	6583	6553	6553
q10	8700	3766	3816	3766
q11	496	252	246	246
q12	395	222	218	218
q13	17752	2989	2990	2989
q14	275	234	235	234
q15	522	488	471	471
q16	507	404	384	384
q17	970	640	627	627
q18	8126	7607	7355	7355
q19	5930	1096	1015	1015
q20	655	315	332	315
q21	4947	3171	3304	3171
q22	354	297	291	291
Total cold run time: 120005 ms
Total hot run time: 39404 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4593	4228	4241	4228
q2	394	279	273	273
q3	2988	2798	2737	2737
q4	1885	1633	1634	1633
q5	5294	5345	5306	5306
q6	215	130	128	128
q7	2187	1754	1734	1734
q8	3217	3397	3340	3340
q9	8459	8417	8372	8372
q10	3921	3659	3660	3659
q11	582	490	499	490
q12	793	623	593	593
q13	16508	2977	3016	2977
q14	312	264	276	264
q15	523	470	479	470
q16	471	406	422	406
q17	1758	1471	1455	1455
q18	7622	7544	7493	7493
q19	1646	1577	1622	1577
q20	1977	1793	1781	1781
q21	4770	4707	4606	4606
q22	596	501	495	495
Total cold run time: 70711 ms
Total hot run time: 54017 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 762e81cafb9c10349402fce13acbabe09b1438b0, data reload: false

query1	900	368	365	365
query2	6457	1933	1813	1813
query3	6654	217	214	214
query4	28087	17332	17507	17332
query5	4170	484	477	477
query6	277	186	188	186
query7	4592	310	293	293
query8	253	199	194	194
query9	8687	2460	2430	2430
query10	442	282	267	267
query11	10874	10180	10058	10058
query12	131	89	86	86
query13	1630	374	369	369
query14	10507	7676	7716	7676
query15	224	165	169	165
query16	7604	478	478	478
query17	1591	550	548	548
query18	1805	283	268	268
query19	189	135	134	134
query20	95	81	83	81
query21	212	95	96	95
query22	4293	3955	3974	3955
query23	33716	33196	33182	33182
query24	12211	2874	2883	2874
query25	670	371	369	369
query26	1837	153	147	147
query27	2908	267	264	264
query28	7252	1996	1986	1986
query29	1109	408	403	403
query30	295	171	151	151
query31	980	742	741	741
query32	97	53	53	53
query33	768	327	328	327
query34	903	473	464	464
query35	865	714	757	714
query36	1091	901	942	901
query37	186	75	79	75
query38	2943	2755	2776	2755
query39	858	808	817	808
query40	272	116	115	115
query41	47	44	45	44
query42	116	93	97	93
query43	523	481	479	479
query44	1179	719	704	704
query45	205	174	175	174
query46	1110	717	732	717
query47	1846	1759	1750	1750
query48	361	293	297	293
query49	1220	418	411	411
query50	806	410	406	406
query51	6845	6664	6647	6647
query52	105	88	86	86
query53	255	178	171	171
query54	1012	443	443	443
query55	73	73	74	73
query56	313	267	275	267
query57	1151	1015	1066	1015
query58	280	269	265	265
query59	3000	2698	2757	2698
query60	315	284	298	284
query61	101	94	123	94
query62	810	637	656	637
query63	212	177	178	177
query64	10572	2220	1720	1720
query65	3160	3124	3102	3102
query66	1348	345	334	334
query67	15417	14900	14659	14659
query68	4574	532	545	532
query69	435	294	310	294
query70	1164	1117	1129	1117
query71	401	271	293	271
query72	6851	5808	5846	5808
query73	749	330	325	325
query74	6152	5630	5640	5630
query75	3495	2687	2661	2661
query76	2830	971	863	863
query77	454	317	313	313
query78	11057	10114	9025	9025
query79	2379	507	514	507
query80	1761	488	476	476
query81	570	220	222	220
query82	593	135	135	135
query83	296	176	173	173
query84	272	80	80	80
query85	1348	315	305	305
query86	487	326	303	303
query87	3271	3123	3054	3054
query88	3966	2465	2450	2450
query89	401	288	278	278
query90	1887	196	194	194
query91	127	99	100	99
query92	65	47	48	47
query93	1733	523	512	512
query94	929	298	298	298
query95	370	274	270	270
query96	596	279	275	275
query97	3190	3033	3029	3029
query98	231	208	198	198
query99	1611	1239	1328	1239
Total cold run time: 285262 ms
Total hot run time: 172537 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 762e81cafb9c10349402fce13acbabe09b1438b0, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.51	0.47	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.56	0.50	0.49
query10	0.54	0.54	0.53
query11	0.15	0.11	0.11
query12	0.16	0.12	0.13
query13	0.59	0.59	0.59
query14	0.76	0.77	0.78
query15	0.86	0.82	0.81
query16	0.37	0.37	0.36
query17	1.03	1.04	1.06
query18	0.22	0.21	0.21
query19	1.77	1.69	1.70
query20	0.02	0.01	0.01
query21	15.42	0.78	0.65
query22	4.35	6.57	2.44
query23	18.30	1.46	1.34
query24	2.11	0.22	0.24
query25	0.16	0.08	0.10
query26	0.29	0.21	0.21
query27	0.46	0.24	0.23
query28	13.23	1.03	1.00
query29	12.60	3.29	3.22
query30	0.28	0.05	0.06
query31	2.87	0.40	0.39
query32	3.26	0.48	0.48
query33	2.88	2.91	2.89
query34	17.00	4.37	4.38
query35	4.41	4.35	4.40
query36	0.65	0.47	0.48
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 31.14 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Gabriel39 @doris-robot