Skip to content

[fix](topn) Fix key topn block reverse is missed in some cases #31199#31200

Merged
xiaokang merged 2 commits intoapache:branch-2.0from
xiaokang:pick_31199
Feb 21, 2024
Merged

[fix](topn) Fix key topn block reverse is missed in some cases #31199#31200
xiaokang merged 2 commits intoapache:branch-2.0from
xiaokang:pick_31199

Conversation

@xiaokang
Copy link
Contributor

Proposed changes

pick from master #31199

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@xiaokang
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}
size_t num_columns = block->columns();
vectorized::IColumn::Permutation permutation;
for (size_t i = 0; i < num_rows; ++i) permutation.emplace_back(num_rows - 1 - i);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: statement should be inside braces [readability-braces-around-statements]

Suggested change
for (size_t i = 0; i < num_rows; ++i) permutation.emplace_back(num_rows - 1 - i);
for (size_t i = 0; i < num_rows; ++i) { permutation.emplace_back(num_rows - 1 - i);
}

vectorized::IColumn::Permutation permutation;
for (size_t i = 0; i < num_rows; ++i) permutation.emplace_back(num_rows - 1 - i);

for (size_t i = 0; i < num_columns; ++i)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: statement should be inside braces [readability-braces-around-statements]

Suggested change
for (size_t i = 0; i < num_columns; ++i)
for (size_t i = 0; i < num_columns; ++i) {

be/src/olap/rowset/segment_v2/segment_iterator.cpp:1994:

-                             block->get_by_position(i).column->permute(permutation, num_rows);
+                             block->get_by_position(i).column->permute(permutation, num_rows);
+ }

@doris-robot
Copy link

TPC-H: Total hot run time: 49855 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f860c14ba390db188a6773641a492b0e41d729ce, data reload: false

------ Round 1 ----------------------------------
q1	17534	4367	4363	4363
q2	2027	151	137	137
q3	10492	1938	1928	1928
q4	10327	1237	1305	1237
q5	8469	4010	4079	4010
q6	238	122	123	122
q7	2009	1583	1613	1583
q8	9538	2686	2706	2686
q9	14004	11328	10443	10443
q10	8646	3480	3469	3469
q11	416	239	243	239
q12	482	294	302	294
q13	18367	3966	3992	3966
q14	357	327	331	327
q15	503	460	460	460
q16	687	584	597	584
q17	1108	921	895	895
q18	7333	6930	6859	6859
q19	1693	1534	1502	1502
q20	524	295	310	295
q21	4383	4065	4112	4065
q22	502	391	396	391
Total cold run time: 119639 ms
Total hot run time: 49855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4370	4319	4319
q2	317	228	225	225
q3	4163	4159	4203	4159
q4	2753	2725	2739	2725
q5	7228	7218	7167	7167
q6	232	120	117	117
q7	3179	2862	2888	2862
q8	4420	4472	4499	4472
q9	17116	16948	17002	16948
q10	4242	4232	4206	4206
q11	748	685	706	685
q12	1025	839	858	839
q13	7063	3793	3744	3744
q14	447	410	416	410
q15	508	455	454	454
q16	762	712	707	707
q17	3853	3824	3836	3824
q18	8798	8757	8751	8751
q19	1695	1665	1671	1665
q20	2363	2140	2131	2131
q21	8521	8478	8542	8478
q22	1059	1001	915	915
Total cold run time: 84858 ms
Total hot run time: 79803 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8035/21247)
Line Coverage: 29.50% (65536/222193)
Region Coverage: 28.96% (33711/116421)
Branch Coverage: 24.81% (17302/69732)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f860c14ba390db188a6773641a492b0e41d729ce_f860c14ba390db188a6773641a492b0e41d729ce/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 240183 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f860c14ba390db188a6773641a492b0e41d729ce, data reload: false

query1	915	384	390	384
query2	6526	2223	2023	2023
query3	6911	208	201	201
query4	20537	18042	17924	17924
query5	19717	6444	6445	6444
query6	291	214	230	214
query7	4141	292	286	286
query8	253	248	241	241
query9	3114	2730	2638	2638
query10	410	281	289	281
query11	11373	10667	10695	10667
query12	121	75	68	68
query13	5579	644	623	623
query14	17387	13231	13513	13231
query15	359	229	239	229
query16	6447	275	259	259
query17	1719	1447	848	848
query18	2354	408	399	399
query19	200	139	151	139
query20	75	72	72	72
query21	185	96	97	96
query22	5337	5209	4996	4996
query23	32509	31847	31941	31847
query24	7012	6448	6426	6426
query25	531	427	434	427
query26	522	157	153	153
query27	1922	293	293	293
query28	6093	2218	2216	2216
query29	2903	2806	2744	2744
query30	240	165	159	159
query31	881	742	739	739
query32	59	60	63	60
query33	397	252	234	234
query34	848	473	484	473
query35	1092	907	882	882
query36	1671	1425	1504	1425
query37	90	62	60	60
query38	3096	2910	2930	2910
query39	1364	1325	1323	1323
query40	207	92	98	92
query41	33	31	31	31
query42	85	82	83	82
query43	694	671	641	641
query44	1124	702	721	702
query45	240	219	227	219
query46	1235	959	972	959
query47	1895	1880	1766	1766
query48	977	678	663	663
query49	614	355	368	355
query50	873	592	645	592
query51	5545	5474	5405	5405
query52	82	80	75	75
query53	449	312	309	309
query54	2639	2426	2477	2426
query55	95	78	82	78
query56	219	205	197	197
query57	1233	1047	1081	1047
query58	216	206	193	193
query59	3456	3249	3168	3168
query60	200	198	211	198
query61	79	81	76	76
query62	799	464	471	464
query63	471	329	323	323
query64	2527	1351	1450	1351
query65	3601	3552	3520	3520
query66	797	363	358	358
query67	18056	15662	16233	15662
query68	8255	658	672	658
query69	552	339	358	339
query70	1927	1741	1654	1654
query71	362	300	307	300
query72	6364	3389	3425	3389
query73	727	322	324	322
query74	6342	5774	5808	5774
query75	4509	3648	3717	3648
query76	4639	1135	1218	1135
query77	607	245	238	238
query78	31989	53965	49474	49474
query79	15636	654	659	654
query80	4883	370	368	368
query81	562	230	226	226
query82	856	93	96	93
query83	438	134	132	132
query84	252	65	66	65
query85	2403	283	269	269
query86	455	376	389	376
query87	3268	2994	3001	2994
query88	6695	2347	2355	2347
query89	422	301	288	288
query90	2507	219	201	201
query91	162	121	122	121
query92	64	51	49	49
query93	5827	597	587	587
query94	1667	203	207	203
query95	1124	1071	1046	1046
query96	654	330	329	329
query97	6520	6311	6421	6311
query98	183	181	178	178
query99	3871	829	910	829
Total cold run time: 348344 ms
Total hot run time: 240183 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f860c14ba390db188a6773641a492b0e41d729ce, data reload: false

query1	0.02	0.02	0.02
query2	0.05	0.03	0.02
query3	0.24	0.04	0.05
query4	1.83	0.08	0.07
query5	0.53	0.53	0.52
query6	1.25	0.61	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.51	0.49	0.48
query10	0.54	0.53	0.53
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.62	0.63	0.62
query14	0.77	0.78	0.81
query15	0.77	0.75	0.75
query16	0.38	0.39	0.36
query17	1.01	1.03	1.05
query18	0.21	0.28	0.22
query19	1.91	1.79	1.86
query20	0.02	0.01	0.00
query21	15.49	0.60	0.56
query22	2.12	2.67	1.33
query23	17.22	1.01	0.92
query24	3.65	3.47	1.32
query25	1.64	0.12	0.12
query26	0.29	0.14	0.14
query27	0.10	0.11	0.11
query28	6.59	0.74	0.76
query29	12.60	2.19	2.32
query30	0.60	0.53	0.53
query31	2.82	0.38	0.40
query32	3.40	0.51	0.49
query33	3.06	3.08	3.07
query34	15.26	4.80	4.77
query35	4.84	4.88	4.84
query36	1.06	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.01	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 102.05 s
Total hot run time: 30.81 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f860c14ba390db188a6773641a492b0e41d729ce with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@xiaokang xiaokang merged commit 89e997c into apache:branch-2.0 Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants