Skip to content

[Bug](top-n) do not update topn filter when sort node and scan node are not in the…#32159

Merged
dataroaring merged 2 commits intoapache:masterfrom
BiteTheDDDDt:fix_0313
Mar 13, 2024
Merged

[Bug](top-n) do not update topn filter when sort node and scan node are not in the…#32159
dataroaring merged 2 commits intoapache:masterfrom
BiteTheDDDDt:fix_0313

Conversation

@BiteTheDDDDt
Copy link
Contributor

… same backend

Proposed changes

do not update topn filter when sort node and scan node are not in the same backend

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37139 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f61fc0471f22da9a7c301a3cac6e65bb9d0ac501, data reload: false

------ Round 1 ----------------------------------
q1	17595	4306	4278	4278
q2	2109	161	149	149
q3	11276	1007	937	937
q4	5031	744	695	695
q5	7872	2761	2809	2761
q6	187	126	126	126
q7	1287	872	847	847
q8	9862	2108	2098	2098
q9	7565	6505	6465	6465
q10	8203	2536	2513	2513
q11	427	228	215	215
q12	772	301	282	282
q13	17992	3026	3007	3007
q14	276	257	248	248
q15	499	458	455	455
q16	475	386	401	386
q17	953	556	601	556
q18	6757	5916	5887	5887
q19	1594	1415	1426	1415
q20	561	278	289	278
q21	7411	3247	3318	3247
q22	804	294	294	294
Total cold run time: 109508 ms
Total hot run time: 37139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4122	4105	4094	4094
q2	323	226	233	226
q3	2955	2865	2850	2850
q4	1839	1565	1532	1532
q5	5221	5240	5224	5224
q6	193	118	117	117
q7	2237	1813	1844	1813
q8	3168	3286	3287	3286
q9	8622	8594	8581	8581
q10	6125	3732	3685	3685
q11	541	453	438	438
q12	722	540	579	540
q13	8864	2879	2878	2878
q14	287	260	259	259
q15	492	444	440	440
q16	453	387	378	378
q17	1717	1492	1471	1471
q18	7736	7146	7169	7146
q19	5652	1543	1504	1504
q20	1922	1691	1688	1688
q21	4890	4772	4698	4698
q22	527	461	462	461
Total cold run time: 68608 ms
Total hot run time: 53309 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186994 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f61fc0471f22da9a7c301a3cac6e65bb9d0ac501, data reload: false

query1	927	355	349	349
query2	7734	2094	1960	1960
query3	6716	217	220	217
query4	31158	20630	20648	20630
query5	4274	418	400	400
query6	270	184	189	184
query7	4626	289	289	289
query8	230	171	169	169
query9	9897	2264	2242	2242
query10	462	239	240	239
query11	17163	14265	14328	14265
query12	133	93	86	86
query13	1632	441	432	432
query14	11363	10706	10610	10610
query15	265	204	194	194
query16	7911	264	254	254
query17	1928	574	547	547
query18	2036	290	277	277
query19	216	159	159	159
query20	94	81	89	81
query21	201	140	126	126
query22	4594	4474	4447	4447
query23	31722	30973	31017	30973
query24	12324	2854	2833	2833
query25	677	397	388	388
query26	1891	160	158	158
query27	3237	354	348	348
query28	8179	1867	1858	1858
query29	1091	653	625	625
query30	309	153	142	142
query31	962	711	733	711
query32	100	60	58	58
query33	771	267	257	257
query34	1103	469	483	469
query35	838	609	606	606
query36	1007	872	902	872
query37	272	77	77	77
query38	3292	3116	3123	3116
query39	1417	1381	1394	1381
query40	298	108	109	108
query41	54	46	45	45
query42	103	99	96	96
query43	488	453	445	445
query44	1305	732	709	709
query45	276	264	250	250
query46	1063	693	691	691
query47	1651	1598	1606	1598
query48	460	355	355	355
query49	1257	358	343	343
query50	765	368	365	365
query51	6898	6653	6777	6653
query52	108	91	121	91
query53	345	280	275	275
query54	332	246	245	245
query55	84	83	81	81
query56	253	240	243	240
query57	1084	1008	1004	1004
query58	239	201	208	201
query59	2692	2627	2618	2618
query60	271	247	266	247
query61	116	113	111	111
query62	595	404	392	392
query63	301	284	276	276
query64	6608	3879	3859	3859
query65	3213	3004	2985	2985
query66	1394	334	316	316
query67	14918	14293	14155	14155
query68	9059	534	538	534
query69	621	400	389	389
query70	1221	1165	1178	1165
query71	530	278	275	275
query72	6552	2671	2488	2488
query73	1192	322	323	322
query74	8025	6530	6451	6451
query75	9577	8057	8015	8015
query76	5379	984	927	927
query77	653	270	269	269
query78	10156	9597	9481	9481
query79	11345	545	530	530
query80	1783	444	390	390
query81	492	210	208	208
query82	316	209	203	203
query83	211	149	146	146
query84	289	78	76	76
query85	1110	323	312	312
query86	363	311	295	295
query87	3410	3311	3212	3212
query88	5345	2294	2300	2294
query89	478	373	410	373
query90	2085	178	177	177
query91	165	128	127	127
query92	60	47	48	47
query93	6711	510	501	501
query94	1329	180	178	178
query95	435	336	337	336
query96	612	270	274	270
query97	4410	4268	4292	4268
query98	229	210	208	208
query99	1058	774	744	744
Total cold run time: 325394 ms
Total hot run time: 186994 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f61fc0471f22da9a7c301a3cac6e65bb9d0ac501, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.11	0.10
query5	0.51	0.51	0.50
query6	1.28	0.63	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.54	0.48	0.50
query10	0.54	0.54	0.56
query11	0.12	0.10	0.09
query12	0.11	0.09	0.08
query13	0.57	0.57	0.57
query14	0.74	0.74	0.77
query15	0.79	0.79	0.77
query16	0.40	0.40	0.38
query17	1.04	0.98	0.94
query18	0.19	0.27	0.23
query19	1.88	1.70	1.84
query20	0.02	0.01	0.01
query21	15.44	0.58	0.56
query22	2.51	2.10	1.90
query23	17.08	0.96	0.77
query24	4.01	2.31	0.83
query25	0.22	0.20	0.04
query26	0.50	0.12	0.12
query27	0.06	0.03	0.03
query28	9.58	0.87	0.85
query29	12.53	3.15	3.20
query30	0.74	0.60	0.60
query31	2.80	0.33	0.34
query32	3.37	0.45	0.44
query33	2.78	2.84	2.77
query34	16.16	4.34	4.26
query35	4.27	4.28	4.32
query36	1.06	1.00	1.00
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.01	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.32 s
Total hot run time: 30.11 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f61fc0471f22da9a7c301a3cac6e65bb9d0ac501 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.6 seconds inserted 10000000 Rows, about 602K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.01% (8574/24488)
Line Coverage: 26.79% (69464/259335)
Region Coverage: 26.05% (36074/138495)
Branch Coverage: 23.00% (18420/80094)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f61fc0471f22da9a7c301a3cac6e65bb9d0ac501_f61fc0471f22da9a7c301a3cac6e65bb9d0ac501/report/index.html

}

bool need_update() const {
std::unique_lock<std::shared_mutex> wlock(_rwlock);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why name wlock? actuall it is a readlock

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 36907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7421d5df58c9fb8085609156d7ddb8a2ac817f3, data reload: false

------ Round 1 ----------------------------------
q1	17698	4285	4119	4119
q2	2027	157	144	144
q3	10635	946	930	930
q4	4640	691	688	688
q5	7596	2761	2774	2761
q6	185	124	126	124
q7	1184	812	808	808
q8	9483	2017	2060	2017
q9	7285	6464	6530	6464
q10	8191	2492	2466	2466
q11	432	223	212	212
q12	764	308	299	299
q13	17966	3015	3023	3015
q14	268	258	257	257
q15	507	462	449	449
q16	499	398	419	398
q17	951	550	559	550
q18	6651	5932	5913	5913
q19	1592	1425	1442	1425
q20	559	280	286	280
q21	7352	3264	3283	3264
q22	814	324	335	324
Total cold run time: 107279 ms
Total hot run time: 36907 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4100	4041	4083	4041
q2	320	218	229	218
q3	2915	2871	2812	2812
q4	1818	1536	1556	1536
q5	5233	5257	5253	5253
q6	213	124	116	116
q7	2240	1891	1821	1821
q8	3159	3272	3286	3272
q9	8524	8487	8553	8487
q10	6123	3664	3668	3664
q11	535	441	448	441
q12	731	534	556	534
q13	11988	2889	2863	2863
q14	286	249	252	249
q15	490	464	433	433
q16	457	421	395	395
q17	1726	1497	1477	1477
q18	7602	7226	7088	7088
q19	3109	1554	1492	1492
q20	1916	1706	1720	1706
q21	4803	4556	4752	4556
q22	570	463	465	463
Total cold run time: 68858 ms
Total hot run time: 52917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185899 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7421d5df58c9fb8085609156d7ddb8a2ac817f3, data reload: false

query1	917	369	344	344
query2	7614	1984	1974	1974
query3	6706	217	216	216
query4	30873	20614	20680	20614
query5	4345	473	412	412
query6	259	177	179	177
query7	4624	306	287	287
query8	228	168	162	162
query9	9903	2240	2235	2235
query10	438	231	240	231
query11	14646	14180	14105	14105
query12	123	82	85	82
query13	1625	419	421	419
query14	11963	10117	10550	10117
query15	331	198	194	194
query16	8164	273	276	273
query17	2026	564	557	557
query18	2122	281	289	281
query19	335	163	151	151
query20	90	88	83	83
query21	201	131	126	126
query22	4631	4430	4510	4430
query23	31717	30839	31307	30839
query24	10648	2844	2819	2819
query25	635	375	378	375
query26	1229	161	159	159
query27	2676	368	358	358
query28	7708	1859	1881	1859
query29	889	624	627	624
query30	307	153	149	149
query31	969	724	717	717
query32	97	64	54	54
query33	775	257	252	252
query34	1056	478	484	478
query35	830	599	598	598
query36	1019	872	852	852
query37	133	79	75	75
query38	3232	3149	3118	3118
query39	1418	1371	1383	1371
query40	207	147	111	111
query41	48	47	46	46
query42	105	99	99	99
query43	479	458	466	458
query44	1152	691	697	691
query45	260	247	260	247
query46	1091	668	692	668
query47	1665	1585	1588	1585
query48	440	344	354	344
query49	1107	335	333	333
query50	754	377	372	372
query51	6869	6781	6657	6657
query52	98	94	87	87
query53	340	272	270	270
query54	302	228	239	228
query55	86	77	85	77
query56	250	236	232	232
query57	1101	1020	1008	1008
query58	229	212	210	210
query59	2791	2629	2786	2629
query60	263	249	290	249
query61	96	91	89	89
query62	595	387	388	387
query63	298	267	270	267
query64	5515	3840	3653	3653
query65	3230	3034	2982	2982
query66	881	323	324	323
query67	14979	14488	14157	14157
query68	6779	533	534	533
query69	629	374	380	374
query70	1222	1150	1154	1150
query71	496	268	263	263
query72	6421	2643	2495	2495
query73	714	319	318	318
query74	7882	6525	6621	6525
query75	9194	8058	8017	8017
query76	4473	793	930	793
query77	629	258	276	258
query78	10347	9636	9653	9636
query79	8205	519	524	519
query80	1565	390	372	372
query81	558	207	213	207
query82	1595	205	209	205
query83	201	146	143	143
query84	288	78	78	78
query85	1514	362	376	362
query86	463	296	327	296
query87	3396	3186	3220	3186
query88	4759	2299	2303	2299
query89	545	372	364	364
query90	2012	186	181	181
query91	170	141	157	141
query92	67	48	48	48
query93	6279	512	493	493
query94	1145	186	188	186
query95	434	339	343	339
query96	615	275	269	269
query97	4427	4270	4260	4260
query98	234	211	208	208
query99	1096	768	755	755
Total cold run time: 311087 ms
Total hot run time: 185899 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7421d5df58c9fb8085609156d7ddb8a2ac817f3, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.51	0.48	0.51
query6	1.23	0.63	0.62
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.54	0.50	0.48
query10	0.54	0.53	0.56
query11	0.11	0.09	0.10
query12	0.11	0.09	0.09
query13	0.58	0.56	0.57
query14	0.72	0.75	0.77
query15	0.79	0.79	0.76
query16	0.37	0.40	0.39
query17	1.01	1.01	1.00
query18	0.20	0.26	0.23
query19	1.88	1.72	1.67
query20	0.01	0.00	0.00
query21	15.40	0.58	0.59
query22	2.64	1.57	1.62
query23	17.19	0.87	0.79
query24	3.49	2.63	1.00
query25	0.10	0.14	0.11
query26	0.77	0.12	0.12
query27	0.03	0.02	0.02
query28	9.71	0.91	0.86
query29	12.51	3.22	3.26
query30	0.64	0.59	0.58
query31	2.80	0.34	0.33
query32	3.41	0.44	0.45
query33	2.82	2.83	2.79
query34	16.13	4.34	4.30
query35	4.35	4.33	4.33
query36	1.09	1.01	1.00
query37	0.07	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.02	0.02	0.03
Total cold run time: 104.19 s
Total hot run time: 30.24 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c7421d5df58c9fb8085609156d7ddb8a2ac817f3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.00% (8574/24496)
Line Coverage: 26.76% (69460/259566)
Region Coverage: 26.03% (36066/138574)
Branch Coverage: 22.97% (18410/80136)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c7421d5df58c9fb8085609156d7ddb8a2ac817f3_c7421d5df58c9fb8085609156d7ddb8a2ac817f3/report/index.html

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 99680e9 into apache:master Mar 13, 2024
yiguolei pushed a commit that referenced this pull request Mar 13, 2024
@BiteTheDDDDt BiteTheDDDDt deleted the fix_0313 branch January 20, 2025 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants