Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids): fix offset in PlanTranslator #29789

Merged
merged 3 commits into from
Jan 15, 2024
Merged

[fix](Nereids): fix offset in PlanTranslator #29789

merged 3 commits into from
Jan 15, 2024

Conversation

jackwener
Copy link
Member

Proposed changes

Current BE operator don't support offset, we need add offset into ExchangeNode

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@jackwener
Copy link
Member Author

run buildall

1 similar comment
@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38781 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 744b24131ed93f411d5525bea649e8865fb02639, data reload: false

------ Round 1 ----------------------------------
q1	18012	4971	4987	4971
q2	2043	155	140	140
q3	10688	1173	1189	1173
q4	10248	789	829	789
q5	7794	3157	3145	3145
q6	203	126	127	126
q7	866	504	494	494
q8	9239	2001	2004	2001
q9	7335	6444	6437	6437
q10	8278	3088	3059	3059
q11	418	208	215	208
q12	362	192	200	192
q13	18138	3475	3449	3449
q14	238	213	213	213
q15	557	513	513	513
q16	428	376	382	376
q17	940	556	487	487
q18	7361	6662	6642	6642
q19	1597	1312	1353	1312
q20	578	320	297	297
q21	2798	2450	2504	2450
q22	355	320	307	307
Total cold run time: 108476 ms
Total hot run time: 38781 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5051	5037	5051	5037
q2	317	210	198	198
q3	3334	3314	3315	3314
q4	2236	2242	2248	2242
q5	5790	5785	5770	5770
q6	199	120	123	120
q7	2289	1834	1842	1834
q8	3436	3538	3578	3538
q9	8907	8811	8848	8811
q10	3759	3819	3848	3819
q11	564	421	429	421
q12	782	599	603	599
q13	5873	3232	3221	3221
q14	285	261	274	261
q15	574	521	510	510
q16	507	453	456	453
q17	2047	2061	1969	1969
q18	8843	8285	8210	8210
q19	1600	1629	1610	1610
q20	2141	1935	1924	1924
q21	6062	5739	5727	5727
q22	583	479	488	479
Total cold run time: 65179 ms
Total hot run time: 60067 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 744b24131ed93f411d5525bea649e8865fb02639, data reload: false

query1	931	333	331	331
query2	6698	1865	1769	1769
query3	6712	204	206	204
query4	26299	22302	22283	22283
query5	5597	503	521	503
query6	265	186	179	179
query7	4621	272	264	264
query8	226	213	215	213
query9	9163	2782	2928	2782
query10	477	251	249	249
query11	16129	15324	15325	15324
query12	134	83	78	78
query13	1621	342	335	335
query14	11983	7384	7156	7156
query15	247	199	197	197
query16	6421	288	262	262
query17	1867	491	486	486
query18	1941	272	272	272
query19	286	143	142	142
query20	87	74	76	74
query21	185	90	90	90
query22	5014	4867	4766	4766
query23	32196	31163	31420	31163
query24	12154	2855	2787	2787
query25	567	336	340	336
query26	1804	157	157	157
query27	2842	277	289	277
query28	6918	1919	1910	1910
query29	2027	394	397	394
query30	283	147	148	147
query31	1023	778	790	778
query32	91	65	63	63
query33	750	276	284	276
query34	890	435	440	435
query35	895	769	802	769
query36	1412	1244	1274	1244
query37	191	70	69	69
query38	3385	3288	3320	3288
query39	1326	1283	1281	1281
query40	323	98	92	92
query41	38	35	35	35
query42	103	91	96	91
query43	523	511	507	507
query44	1041	712	728	712
query45	207	187	184	184
query46	1046	682	666	666
query47	1652	1603	1576	1576
query48	385	315	303	303
query49	1259	314	308	308
query50	721	322	351	322
query51	5282	5231	5254	5231
query52	111	83	98	83
query53	216	155	145	145
query54	1413	580	610	580
query55	94	90	92	90
query56	219	201	191	191
query57	1029	951	938	938
query58	246	212	215	212
query59	2798	2556	2520	2520
query60	252	215	224	215
query61	83	83	81	81
query62	628	463	436	436
query63	173	151	156	151
query64	5816	1738	1666	1666
query65	3341	3257	3261	3257
query66	1338	345	332	332
query67	15298	15186	15126	15126
query68	12586	517	540	517
query69	550	279	282	279
query70	1736	1527	1545	1527
query71	502	219	223	219
query72	4882	2876	2851	2851
query73	2610	309	321	309
query74	7031	6506	6496	6496
query75	4984	2244	2275	2244
query76	6348	915	1066	915
query77	673	278	278	278
query78	9789	8702	8664	8664
query79	1035	510	503	503
query80	557	341	362	341
query81	447	208	212	208
query82	212	93	87	87
query83	155	141	142	141
query84	266	56	55	55
query85	1003	295	293	293
query86	382	375	383	375
query87	3619	3430	3401	3401
query88	3033	2230	2220	2220
query89	341	267	250	250
query90	1902	192	194	192
query91	164	137	130	130
query92	60	57	60	57
query93	1558	423	475	423
query94	853	195	194	194
query95	479	433	415	415
query96	620	316	317	316
query97	4309	4119	4228	4119
query98	197	193	183	183
query99	1118	880	846	846
Total cold run time: 295294 ms
Total hot run time: 178431 ms

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38738 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f3d55e3bdea7d73fc16e74569ff756cef89b49cf, data reload: false

------ Round 1 ----------------------------------
q1	18138	5007	5017	5007
q2	2062	145	138	138
q3	10792	1145	1173	1145
q4	10326	797	832	797
q5	7811	3137	3162	3137
q6	203	128	125	125
q7	884	512	496	496
q8	9259	1942	1988	1942
q9	7337	6496	6426	6426
q10	8257	3105	3057	3057
q11	407	217	212	212
q12	356	198	197	197
q13	18098	3446	3440	3440
q14	249	217	220	217
q15	556	515	519	515
q16	433	389	388	388
q17	949	505	537	505
q18	7280	6728	6690	6690
q19	1578	1344	1416	1344
q20	553	323	311	311
q21	2761	2345	2395	2345
q22	345	311	304	304
Total cold run time: 108634 ms
Total hot run time: 38738 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4961	4950	4997	4950
q2	319	202	207	202
q3	3331	3323	3323	3323
q4	2246	2269	2266	2266
q5	5812	5793	5815	5793
q6	192	119	120	119
q7	2310	1849	1845	1845
q8	3446	3547	3562	3547
q9	8880	8797	8836	8797
q10	3747	3838	3817	3817
q11	565	425	437	425
q12	785	606	604	604
q13	5213	3210	3204	3204
q14	294	269	274	269
q15	574	525	512	512
q16	525	470	485	470
q17	2040	2049	2002	2002
q18	8796	8321	8311	8311
q19	1637	1586	1602	1586
q20	2158	1955	1966	1955
q21	6035	5744	5795	5744
q22	578	448	456	448
Total cold run time: 64444 ms
Total hot run time: 60189 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 177745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f3d55e3bdea7d73fc16e74569ff756cef89b49cf, data reload: false

query1	924	339	333	333
query2	6710	1865	1822	1822
query3	6703	220	209	209
query4	26027	22176	22034	22034
query5	5665	500	504	500
query6	270	179	191	179
query7	4629	288	270	270
query8	227	201	198	198
query9	9195	2799	2726	2726
query10	573	248	233	233
query11	16139	15492	15323	15323
query12	131	81	77	77
query13	1617	356	347	347
query14	11917	7216	7197	7197
query15	266	185	194	185
query16	6418	288	276	276
query17	1843	501	479	479
query18	1952	285	272	272
query19	293	146	144	144
query20	85	79	81	79
query21	190	89	86	86
query22	5002	4870	4740	4740
query23	31716	31256	30955	30955
query24	12437	2845	2873	2845
query25	576	344	338	338
query26	1794	160	154	154
query27	2898	276	285	276
query28	7246	1850	1842	1842
query29	2078	413	403	403
query30	288	143	145	143
query31	971	766	770	766
query32	89	64	58	58
query33	759	284	270	270
query34	1059	462	454	454
query35	935	786	758	758
query36	1405	1230	1264	1230
query37	188	73	70	70
query38	3420	3293	3311	3293
query39	1329	1276	1296	1276
query40	324	105	94	94
query41	42	36	36	36
query42	101	96	95	95
query43	566	513	510	510
query44	1092	703	743	703
query45	209	186	185	185
query46	1131	674	664	664
query47	1572	1557	1540	1540
query48	410	321	324	321
query49	1278	322	328	322
query50	842	339	335	335
query51	5406	5255	5221	5221
query52	93	92	91	91
query53	227	165	156	156
query54	1452	571	592	571
query55	99	91	94	91
query56	221	202	198	198
query57	1026	945	939	939
query58	231	215	217	215
query59	2742	2546	2474	2474
query60	255	236	220	220
query61	84	83	84	83
query62	574	410	381	381
query63	174	157	153	153
query64	5847	1742	1727	1727
query65	3332	3260	3242	3242
query66	1323	343	318	318
query67	15652	15550	14900	14900
query68	10759	548	545	545
query69	577	289	280	280
query70	1704	1573	1546	1546
query71	506	213	208	208
query72	4764	2832	2860	2832
query73	2112	315	315	315
query74	6985	6573	6475	6475
query75	4811	2265	2289	2265
query76	6262	947	1265	947
query77	669	298	263	263
query78	9649	8667	8637	8637
query79	1116	520	513	513
query80	564	357	364	357
query81	448	214	213	213
query82	227	98	94	94
query83	169	146	135	135
query84	266	59	61	59
query85	965	300	292	292
query86	394	381	412	381
query87	3604	3398	3426	3398
query88	3252	2255	2244	2244
query89	357	262	267	262
query90	1821	206	203	203
query91	172	134	142	134
query92	64	56	56	56
query93	2006	445	442	442
query94	901	198	208	198
query95	498	438	435	435
query96	680	325	325	325
query97	4283	4165	4193	4165
query98	197	198	185	185
query99	987	718	716	716
Total cold run time: 294038 ms
Total hot run time: 177745 ms

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e01e0e046a7d78ce3a138b32dfa83647caa41913, data reload: false

------ Round 1 ----------------------------------
q1	17990	5823	5884	5823
q2	2041	146	140	140
q3	10909	1167	1186	1167
q4	10575	793	789	789
q5	7834	3177	3198	3177
q6	210	124	123	123
q7	893	494	494	494
q8	9293	1999	2000	1999
q9	7317	6416	6398	6398
q10	8239	3111	3117	3111
q11	409	211	212	211
q12	362	200	194	194
q13	18101	3412	3401	3401
q14	261	220	215	215
q15	554	522	510	510
q16	432	375	376	375
q17	946	514	562	514
q18	7548	6870	6882	6870
q19	1603	1400	1363	1363
q20	570	295	269	269
q21	2825	2479	2502	2479
q22	390	335	320	320
Total cold run time: 109302 ms
Total hot run time: 39942 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5764	5725	5606	5606
q2	353	220	224	220
q3	3349	3224	3154	3154
q4	2237	2078	2125	2078
q5	6107	6015	5928	5928
q6	203	122	123	122
q7	2371	1949	1879	1879
q8	3582	3671	3725	3671
q9	9543	22325	9502	9502
q10	7060	3735	3836	3735
q11	583	474	463	463
q12	1246	658	636	636
q13	18174	3393	3280	3280
q14	425	259	271	259
q15	582	527	523	523
q16	535	481	482	481
q17	9173	1985	1979	1979
q18	9688	9818	11880	9818
q19	12994	1655	1652	1652
q20	5722	2063	2022	2022
q21	14217	5785	6362	5785
q22	3295	730	894	730
Total cold run time: 117203 ms
Total hot run time: 63523 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178254 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e01e0e046a7d78ce3a138b32dfa83647caa41913, data reload: false

query1	2742	343	328	328
query2	7798	1823	1810	1810
query3	6720	204	205	204
query4	33654	22102	22163	22102
query5	6926	587	574	574
query6	252	173	169	169
query7	4634	280	265	265
query8	228	192	205	192
query9	9180	2697	2661	2661
query10	651	246	247	246
query11	16329	15341	15249	15249
query12	130	69	71	69
query13	1715	396	374	374
query14	11331	7390	7321	7321
query15	214	180	180	180
query16	4614	259	243	243
query17	1005	490	478	478
query18	1799	274	258	258
query19	188	148	149	148
query20	80	74	75	74
query21	198	132	133	132
query22	4943	4811	4782	4782
query23	31733	31009	31071	31009
query24	7760	2743	2792	2743
query25	547	324	323	323
query26	890	156	154	154
query27	2828	284	273	273
query28	4962	1855	1857	1855
query29	1035	689	713	689
query30	278	137	137	137
query31	921	743	757	743
query32	73	59	55	55
query33	461	245	232	232
query34	820	468	468	468
query35	846	787	788	787
query36	1424	1305	1233	1233
query37	102	66	70	66
query38	3386	3338	3255	3255
query39	1327	1264	1266	1264
query40	218	103	92	92
query41	39	36	35	35
query42	100	92	97	92
query43	547	499	519	499
query44	1055	705	732	705
query45	204	179	184	179
query46	1031	629	653	629
query47	1619	1653	1611	1611
query48	430	329	320	320
query49	1131	308	325	308
query50	704	320	318	318
query51	5339	5171	5276	5171
query52	103	87	94	87
query53	360	290	291	290
query54	764	473	460	460
query55	94	92	88	88
query56	201	194	191	191
query57	999	963	960	960
query58	204	190	196	190
query59	2956	2665	2647	2647
query60	242	226	209	209
query61	82	83	81	81
query62	638	410	395	395
query63	314	290	287	287
query64	4167	1690	1629	1629
query65	3370	3273	3240	3240
query66	1280	334	327	327
query67	15811	15002	15334	15002
query68	11642	533	530	530
query69	618	333	330	330
query70	1724	1559	1536	1536
query71	499	233	243	233
query72	5177	2810	2861	2810
query73	2322	315	313	313
query74	7423	6403	6471	6403
query75	4924	2340	2304	2304
query76	6057	967	1040	967
query77	679	274	258	258
query78	9100	8612	8775	8612
query79	1024	534	505	505
query80	564	347	354	347
query81	470	207	204	204
query82	213	93	95	93
query83	144	121	121	121
query84	283	71	80	71
query85	1066	330	321	321
query86	410	356	393	356
query87	3580	3421	3334	3334
query88	2913	2205	2228	2205
query89	449	385	376	376
query90	2048	206	198	198
query91	155	132	132	132
query92	68	58	53	53
query93	1569	448	421	421
query94	1254	180	181	180
query95	521	482	487	482
query96	618	312	318	312
query97	4265	4146	4158	4146
query98	204	179	188	179
query99	1019	736	735	735
Total cold run time: 291728 ms
Total hot run time: 178254 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.51 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 22.2 seconds inserted 10000000 Rows, about 450K ops/s
storage size: 17183638280 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e01e0e046a7d78ce3a138b32dfa83647caa41913, data reload: false

query1	0.06	0.06	0.05
query2	0.07	0.03	0.02
query3	0.25	0.12	0.11
query4	1.74	0.13	0.12
query5	0.54	0.53	0.54
query6	1.37	0.64	0.64
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.51
query10	0.58	0.57	0.56
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.60	0.60	0.61
query14	0.77	0.79	0.77
query15	0.79	0.78	0.77
query16	0.37	0.39	0.36
query17	1.02	1.03	1.01
query18	0.25	0.24	0.25
query19	1.85	1.76	1.80
query20	0.01	0.01	0.02
query21	15.40	0.58	0.55
query22	2.69	2.42	2.13
query23	17.24	0.91	0.76
query24	16.22	0.60	0.59
query25	2.15	0.15	0.14
query26	0.14	0.13	0.13
query27	0.14	0.16	0.14
query28	6.49	0.83	0.89
query29	12.55	3.29	3.26
query30	0.59	0.49	0.49
query31	2.78	0.34	0.34
query32	3.38	0.48	0.48
query33	3.21	3.27	3.20
query34	15.82	4.31	4.23
query35	4.25	4.25	4.23
query36	1.10	1.07	1.08
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.14	0.15
query41	0.08	0.01	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 115.71 s
Total hot run time: 31.14 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e01e0e046a7d78ce3a138b32dfa83647caa41913 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@jackwener
fix
1a7f368 
[fix](Nereids): fix offset in PlanTranslator
@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14f1fb76ce2201a223814f7cf89792da2e647eda, data reload: false

------ Round 1 ----------------------------------
q1	7484	5679	5798	5679
q2	1627	147	138	138
q3	1503	1232	1231	1231
q4	9782	817	822	817
q5	7829	3252	3162	3162
q6	206	126	127	126
q7	888	493	480	480
q8	9322	1994	2019	1994
q9	7311	6458	6403	6403
q10	8311	3067	3081	3067
q11	425	215	211	211
q12	363	201	199	199
q13	18103	3443	3414	3414
q14	256	211	226	211
q15	552	526	507	507
q16	425	378	389	378
q17	968	509	502	502
q18	7564	7006	7115	7006
q19	1617	1360	1420	1360
q20	540	312	288	288
q21	2846	2484	2447	2447
q22	379	319	314	314
Total cold run time: 88301 ms
Total hot run time: 39934 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5785	5848	5804	5804
q2	342	233	227	227
q3	3410	3160	3082	3082
q4	2264	2072	2044	2044
q5	6091	5901	6112	5901
q6	206	123	124	123
q7	2404	1911	1997	1911
q8	3531	3873	3706	3706
q9	9787	27364	9506	9506
q10	4960	3835	3856	3835
q11	616	475	460	460
q12	1408	678	649	649
q13	17107	3317	3253	3253
q14	338	252	273	252
q15	602	515	514	514
q16	539	483	488	483
q17	9372	2072	1940	1940
q18	9495	9575	13680	9575
q19	13028	1522	1598	1522
q20	4225	1955	1934	1934
q21	13611	5640	5899	5640
q22	1095	606	659	606
Total cold run time: 110216 ms
Total hot run time: 62967 ms

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39520 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a7f3686d8ce803ed7b7652d822511216e45ff60, data reload: false

------ Round 1 ----------------------------------
q1	17981	6089	5835	5835
q2	2037	141	135	135
q3	10914	1116	1121	1116
q4	10558	801	805	801
q5	7859	3170	3200	3170
q6	209	123	123	123
q7	869	486	490	486
q8	9257	1923	1998	1923
q9	7323	6426	6363	6363
q10	8213	3057	3077	3057
q11	407	210	204	204
q12	368	192	195	192
q13	18135	3430	3406	3406
q14	248	222	227	222
q15	550	511	509	509
q16	437	378	390	378
q17	932	466	519	466
q18	7661	6956	6790	6790
q19	1586	1320	1427	1320
q20	552	307	311	307
q21	2737	2433	2413	2413
q22	377	315	304	304
Total cold run time: 109210 ms
Total hot run time: 39520 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5714	5711	6006	5711
q2	342	227	211	211
q3	3392	3120	3126	3120
q4	2195	2092	2000	2000
q5	6096	6032	6037	6032
q6	204	123	126	123
q7	2395	1996	1994	1994
q8	3663	3679	3726	3679
q9	9540	20001	9505	9505
q10	8210	3750	3846	3750
q11	586	464	477	464
q12	1212	689	678	678
q13	17085	3270	3240	3240
q14	299	251	260	251
q15	576	508	514	508
q16	547	472	497	472
q17	2437	1904	2001	1904
q18	9518	9556	14855	9556
q19	13434	1652	1649	1649
q20	4079	1939	1930	1930
q21	14593	5744	5846	5744
q22	2383	608	590	590
Total cold run time: 108500 ms
Total hot run time: 63111 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178517 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a7f3686d8ce803ed7b7652d822511216e45ff60, data reload: false

query1	930	335	329	329
query2	6735	1931	1940	1931
query3	6715	217	210	210
query4	37928	22109	22236	22109
query5	6955	554	572	554
query6	258	186	199	186
query7	4629	270	276	270
query8	236	199	216	199
query9	9228	2787	2734	2734
query10	562	245	262	245
query11	16848	15317	15311	15311
query12	126	72	70	70
query13	1712	373	379	373
query14	11339	7355	7432	7355
query15	217	184	186	184
query16	4639	259	246	246
query17	991	491	476	476
query18	1809	267	268	267
query19	178	151	146	146
query20	81	81	80	80
query21	199	135	131	131
query22	4994	4665	4713	4665
query23	31960	31153	30964	30964
query24	12440	2828	2691	2691
query25	567	320	316	316
query26	1755	148	154	148
query27	3184	285	277	277
query28	7121	1846	1819	1819
query29	2213	688	697	688
query30	295	141	138	138
query31	928	753	767	753
query32	77	60	63	60
query33	713	261	237	237
query34	1150	479	471	471
query35	891	776	759	759
query36	1440	1235	1245	1235
query37	96	72	69	69
query38	3372	3245	3252	3245
query39	1352	1285	1271	1271
query40	344	100	92	92
query41	39	35	38	35
query42	100	93	101	93
query43	540	506	500	500
query44	1114	699	710	699
query45	203	197	184	184
query46	1078	655	655	655
query47	1693	1565	1616	1565
query48	419	320	330	320
query49	1256	311	298	298
query50	691	319	312	312
query51	5313	5202	5265	5202
query52	104	93	96	93
query53	356	306	282	282
query54	876	453	459	453
query55	94	96	86	86
query56	226	203	204	203
query57	1025	953	956	953
query58	215	196	188	188
query59	2978	2727	2845	2727
query60	268	233	210	210
query61	85	93	79	79
query62	674	419	401	401
query63	309	295	284	284
query64	6269	1663	1658	1658
query65	3356	3282	3295	3282
query66	1429	334	338	334
query67	15542	15030	15215	15030
query68	11158	516	516	516
query69	643	344	343	343
query70	1653	1640	1564	1564
query71	560	233	259	233
query72	4825	2824	2833	2824
query73	2069	326	324	324
query74	7116	6415	6409	6409
query75	4722	2394	2284	2284
query76	6343	966	1058	966
query77	675	258	292	258
query78	9076	8711	8487	8487
query79	1032	509	519	509
query80	529	345	360	345
query81	441	204	207	204
query82	210	90	89	89
query83	143	126	156	126
query84	277	73	71	71
query85	1056	341	333	333
query86	377	369	384	369
query87	3537	3379	3361	3361
query88	3170	2276	2303	2276
query89	462	412	390	390
query90	2006	221	198	198
query91	156	132	121	121
query92	64	56	54	54
query93	2515	411	408	408
query94	1221	190	181	181
query95	528	489	480	480
query96	640	326	321	321
query97	4275	4120	4168	4120
query98	200	193	189	189
query99	1045	748	734	734
Total cold run time: 306183 ms
Total hot run time: 178517 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.72 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 22.4 seconds inserted 10000000 Rows, about 446K ops/s
storage size: 17183445129 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a7f3686d8ce803ed7b7652d822511216e45ff60, data reload: false

query1	0.06	0.06	0.05
query2	0.06	0.02	0.02
query3	0.26	0.13	0.11
query4	1.76	0.12	0.11
query5	0.54	0.53	0.54
query6	1.32	0.63	0.65
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.54	0.51	0.49
query10	0.55	0.56	0.54
query11	0.11	0.09	0.09
query12	0.12	0.10	0.09
query13	0.61	0.59	0.60
query14	0.77	0.79	0.79
query15	0.79	0.78	0.78
query16	0.37	0.39	0.38
query17	1.02	1.02	0.99
query18	0.23	0.26	0.24
query19	1.86	1.79	1.85
query20	0.01	0.01	0.01
query21	15.44	0.61	0.57
query22	2.41	3.28	2.07
query23	17.40	0.80	0.80
query24	2.42	1.40	0.45
query25	1.94	0.16	0.14
query26	0.14	0.14	0.13
query27	0.15	0.15	0.14
query28	11.00	0.80	0.81
query29	12.62	3.24	3.14
query30	0.58	0.52	0.47
query31	2.78	0.34	0.35
query32	3.40	0.48	0.48
query33	3.24	3.22	3.21
query34	15.98	4.29	4.22
query35	4.24	4.31	4.27
query36	1.12	1.09	1.07
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 106.28 s
Total hot run time: 30.82 s

@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39937 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a65d3fd9601e8c9f20a69149aede7481ffabbe2a, data reload: false

------ Round 1 ----------------------------------
q1	18008	5988	5905	5905
q2	2044	141	138	138
q3	10903	1171	1174	1171
q4	10545	818	836	818
q5	7851	3179	3160	3160
q6	208	126	127	126
q7	880	511	503	503
q8	9284	1976	1988	1976
q9	7326	6389	6433	6389
q10	8210	3106	3030	3030
q11	409	218	207	207
q12	369	201	203	201
q13	18119	3406	3425	3406
q14	251	214	222	214
q15	564	520	505	505
q16	445	399	384	384
q17	941	509	591	509
q18	7661	6899	6846	6846
q19	1618	1380	1367	1367
q20	575	289	317	289
q21	2886	2499	2466	2466
q22	378	329	327	327
Total cold run time: 109475 ms
Total hot run time: 39937 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5795	5821	5821	5821
q2	360	223	227	223
q3	3331	3160	3196	3160
q4	2254	2103	2039	2039
q5	6090	6043	5984	5984
q6	202	118	123	118
q7	2330	1957	2006	1957
q8	3645	3667	3733	3667
q9	9634	24005	9498	9498
q10	9734	3719	3809	3719
q11	647	464	451	451
q12	1196	666	645	645
q13	17122	3291	3255	3255
q14	376	259	265	259
q15	567	503	511	503
q16	547	477	463	463
q17	8725	1996	1958	1958
q18	9601	9525	11195	9525
q19	13813	1634	1603	1603
q20	3939	1963	1944	1944
q21	14458	5588	5731	5588
q22	1006	607	604	604
Total cold run time: 115372 ms
Total hot run time: 62984 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178817 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a65d3fd9601e8c9f20a69149aede7481ffabbe2a, data reload: false

query1	939	331	328	328
query2	6759	1881	1877	1877
query3	6717	205	205	205
query4	36099	22378	22092	22092
query5	6958	580	531	531
query6	270	180	178	178
query7	4614	283	276	276
query8	235	192	202	192
query9	9153	2767	2844	2767
query10	438	238	247	238
query11	16950	15468	15252	15252
query12	129	72	73	72
query13	1691	375	373	373
query14	11379	7472	7418	7418
query15	222	183	183	183
query16	4637	249	240	240
query17	1019	487	489	487
query18	1803	265	262	262
query19	180	147	153	147
query20	89	77	73	73
query21	192	139	136	136
query22	5033	4777	4793	4777
query23	31837	30985	31094	30985
query24	7566	2801	2794	2794
query25	411	330	320	320
query26	725	148	152	148
query27	2332	285	269	269
query28	3806	1891	1867	1867
query29	973	691	679	679
query30	204	139	138	138
query31	947	766	794	766
query32	69	61	59	59
query33	435	235	233	233
query34	843	460	457	457
query35	896	789	790	789
query36	1436	1221	1259	1221
query37	100	68	73	68
query38	3427	3324	3309	3309
query39	1330	1270	1285	1270
query40	208	98	96	96
query41	38	35	36	35
query42	101	93	92	92
query43	555	489	477	477
query44	1042	704	725	704
query45	206	184	182	182
query46	1038	641	623	623
query47	1700	1638	1652	1638
query48	402	323	308	308
query49	629	318	316	316
query50	707	309	308	308
query51	5339	5193	5272	5193
query52	105	87	94	87
query53	360	276	289	276
query54	724	453	458	453
query55	95	91	89	89
query56	207	192	196	192
query57	978	971	947	947
query58	205	197	191	191
query59	3035	2727	2789	2727
query60	244	234	226	226
query61	84	86	83	83
query62	495	414	420	414
query63	303	288	278	278
query64	2588	1722	1669	1669
query65	3380	3272	3296	3272
query66	1063	336	331	331
query67	15695	15068	14899	14899
query68	10271	506	521	506
query69	621	332	344	332
query70	1678	1584	1611	1584
query71	501	239	229	229
query72	4991	2860	2837	2837
query73	2167	315	312	312
query74	7056	6479	6469	6469
query75	4729	2369	2305	2305
query76	6106	949	1046	949
query77	697	268	285	268
query78	9708	8803	8633	8633
query79	1013	496	498	496
query80	570	353	342	342
query81	474	208	209	208
query82	191	94	89	89
query83	141	124	121	121
query84	286	76	71	71
query85	1052	350	327	327
query86	391	385	359	359
query87	3542	3371	3437	3371
query88	2853	2203	2195	2195
query89	444	377	380	377
query90	2262	211	197	197
query91	167	123	133	123
query92	63	56	53	53
query93	984	408	427	408
query94	1348	178	175	175
query95	529	481	472	472
query96	621	314	315	314
query97	4309	4168	4177	4168
query98	208	197	187	187
query99	1055	758	763	758
Total cold run time: 285397 ms
Total hot run time: 178817 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.95 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s
storage size: 17184284192 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a65d3fd9601e8c9f20a69149aede7481ffabbe2a, data reload: false

query1	0.06	0.06	0.05
query2	0.06	0.03	0.02
query3	0.25	0.12	0.12
query4	1.78	0.13	0.11
query5	0.54	0.52	0.53
query6	1.35	0.62	0.66
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.50	0.51
query10	0.57	0.57	0.57
query11	0.12	0.09	0.08
query12	0.11	0.09	0.08
query13	0.60	0.60	0.60
query14	0.78	0.78	0.80
query15	0.80	0.78	0.78
query16	0.37	0.39	0.37
query17	1.02	1.01	1.02
query18	0.26	0.25	0.25
query19	1.87	1.81	1.77
query20	0.01	0.01	0.02
query21	15.40	0.57	0.55
query22	2.93	1.79	1.52
query23	17.03	0.85	0.77
query24	2.43	1.16	1.04
query25	0.36	0.22	0.06
query26	0.62	0.14	0.14
query27	0.07	0.06	0.07
query28	11.55	0.79	0.80
query29	12.58	3.23	3.18
query30	0.56	0.51	0.50
query31	2.78	0.34	0.35
query32	3.40	0.48	0.48
query33	3.23	3.22	3.23
query34	15.76	4.23	4.23
query35	4.28	4.27	4.23
query36	1.12	1.07	1.07
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.17	0.13	0.12
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.66 s
Total hot run time: 30.69 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a65d3fd9601e8c9f20a69149aede7481ffabbe2a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.8 seconds inserted 10000000 Rows, about 675K ops/s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@jackwener jackwener merged commit 0882223 into apache:master Jan 15, 2024
26 of 28 checks passed
@jackwener jackwener deleted the fix_limit branch January 15, 2024 09:34
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Jan 16, 2024
@jackwener @Doris-Extras
[fix](Nereids): fix offset in PlanTranslator (#29789)
0ea5f0f 
Current BE operator don't support `offset`, we need add offset into `ExchangeNode`
@morrySnow morrySnow added the p0_w label Jan 31, 2024
jackwener added a commit to jackwener/doris that referenced this pull request Feb 1, 2024
@jackwener
[fix](Nereids): fix offset in PlanTranslator (apache#29789)
8eab287 
Current BE operator don't support `offset`, we need add offset into `ExchangeNode`

(cherry picked from commit 0882223)
@jackwener jackwener mentioned this pull request Feb 1, 2024
Merged
jackwener added a commit that referenced this pull request Feb 1, 2024
@jackwener
[fix](Nereids): fix offset in PlanTranslator (#29789) (#30684)
7704223 
* [fix](Nereids): fix offset in PlanTranslator (#29789)

Current BE operator don't support `offset`, we need add offset into `ExchangeNode`

(cherry picked from commit 0882223)

* fix compile
wsjz pushed a commit to wsjz/incubator-doris that referenced this pull request Feb 4, 2024
@jackwener @wsjz
[fix](Nereids): fix offset in PlanTranslator (apache#29789)
552f304 
Current BE operator don't support `offset`, we need add offset into `ExchangeNode`
@morrySnow morrySnow mentioned this pull request Aug 13, 2024
Merged
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@jackwener
[fix](Nereids): fix offset in PlanTranslator (apache#29789) (apache#3…
450cb33 
…0684)

* [fix](Nereids): fix offset in PlanTranslator (apache#29789)

Current BE operator don't support `offset`, we need add offset into `ExchangeNode`

(cherry picked from commit 0882223)

* fix compile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@keanji-x keanji-x keanji-x approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/3.0.0-merged p0_w reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@jackwener @doris-robot @keanji-x @starocean999 @xiaokang @wm1581066 @dataroaring @morrySnow