Skip to content

[Bug](load) fix load failed on stream load tvf into agg state #28420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 6 commits into from
Jan 4, 2024
Merged

[Bug](load) fix load failed on stream load tvf into agg state #28420

BiteTheDDDDt merged 6 commits into from
Jan 4, 2024

Conversation

@BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented Dec 14, 2023

Proposed changes

fix load failed on stream load tvf into agg state

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 14, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 14, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 335f76e34f2ff2e01ef1e2f652f88de7aef84dff, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4715	4433	4478	4433
q2	366	148	157	148
q3	1482	1273	1210	1210
q4	1127	952	935	935
q5	3166	3200	3169	3169
q6	250	129	132	129
q7	1029	500	477	477
q8	2223	2223	2204	2204
q9	6715	6639	6688	6639
q10	3220	3283	3256	3256
q11	327	197	205	197
q12	346	210	203	203
q13	4575	3835	3793	3793
q14	239	211	208	208
q15	566	528	526	526
q16	436	381	387	381
q17	1010	576	556	556
q18	7233	6933	6847	6847
q19	1526	1426	1428	1426
q20	527	283	309	283
q21	3075	2674	2670	2670
q22	344	274	287	274
Total cold run time: 44497 ms
Total hot run time: 39964 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4373	4637	4382	4382
q2	270	165	174	165
q3	3545	3535	3511	3511
q4	2388	2372	2368	2368
q5	5731	5736	5715	5715
q6	238	120	119	119
q7	2380	1838	1833	1833
q8	3531	3538	3531	3531
q9	9046	9016	9023	9016
q10	3909	3956	3989	3956
q11	504	384	380	380
q12	769	597	592	592
q13	4258	3513	3535	3513
q14	286	268	258	258
q15	576	516	520	516
q16	517	442	481	442
q17	1895	1855	1852	1852
q18	8633	8204	8351	8204
q19	1722	1741	1743	1741
q20	2246	1934	1939	1934
q21	6552	6193	6151	6151
q22	512	414	417	414
Total cold run time: 63881 ms
Total hot run time: 60593 ms

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 15, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 15, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.42 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17221092736 Bytes

@doris-robot
Copy link

doris-robot commented Dec 15, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 714be08583846d8f242b0b6a6dbd9e1ab0a26eb8, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4744	4450	4456	4450
q2	368	148	139	139
q3	1470	1284	1238	1238
q4	1114	910	935	910
q5	3186	3206	3207	3206
q6	243	129	128	128
q7	1004	487	484	484
q8	2211	2225	2209	2209
q9	6672	6714	6652	6652
q10	3230	3273	3289	3273
q11	315	214	202	202
q12	355	209	206	206
q13	4549	3828	3803	3803
q14	244	209	214	209
q15	570	523	519	519
q16	432	381	382	381
q17	1028	689	552	552
q18	7116	6857	6901	6857
q19	1550	1451	1377	1377
q20	559	331	313	313
q21	3072	2694	2711	2694
q22	342	277	286	277
Total cold run time: 44374 ms
Total hot run time: 40079 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4450	4411	4384	4384
q2	270	162	164	162
q3	3495	3510	3493	3493
q4	2391	2374	2369	2369
q5	5747	5725	5725	5725
q6	237	119	119	119
q7	2392	1864	1863	1863
q8	3522	3525	3521	3521
q9	9065	9007	8978	8978
q10	3916	4034	4009	4009
q11	497	373	378	373
q12	767	594	581	581
q13	4312	3541	3565	3541
q14	286	269	244	244
q15	578	527	516	516
q16	502	446	486	446
q17	1891	1876	1862	1862
q18	8602	8164	8195	8164
q19	1745	1738	1770	1738
q20	2256	1988	1913	1913
q21	6497	6137	6102	6102
q22	494	411	423	411
Total cold run time: 63912 ms
Total hot run time: 60514 ms

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 15, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 15, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.18 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17224323725 Bytes

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 16, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 16, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.49 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17220965957 Bytes

@doris-robot
Copy link

doris-robot commented Dec 16, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 1568c530232b0549fc65df2b8e2faa2a2e42e48e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4735	4448	4463	4448
q2	364	149	158	149
q3	1473	1252	1238	1238
q4	1112	879	918	879
q5	3163	3207	3185	3185
q6	247	133	129	129
q7	987	491	481	481
q8	2202	2200	2168	2168
q9	6675	6653	6672	6653
q10	3216	3270	3266	3266
q11	314	201	202	201
q12	352	211	204	204
q13	4575	3789	3830	3789
q14	251	207	215	207
q15	575	535	531	531
q16	437	387	387	387
q17	1030	601	616	601
q18	7033	6930	6757	6757
q19	1538	1458	1317	1317
q20	531	336	319	319
q21	3053	2657	2637	2637
q22	352	282	285	282
Total cold run time: 44215 ms
Total hot run time: 39828 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4449	4393	4434	4393
q2	267	163	174	163
q3	3515	3498	3499	3498
q4	2373	2375	2375	2375
q5	5741	5733	5758	5733
q6	239	123	125	123
q7	2361	1859	1841	1841
q8	3514	3526	3533	3526
q9	8997	8957	9004	8957
q10	3932	4019	4044	4019
q11	501	388	396	388
q12	768	597	607	597
q13	4342	3562	3534	3534
q14	286	266	262	262
q15	564	518	527	518
q16	501	474	467	467
q17	1884	1882	1860	1860
q18	8535	8106	8087	8087
q19	1726	1721	1703	1703
q20	2256	1941	1923	1923
q21	6502	6092	6128	6092
q22	488	429	423	423
Total cold run time: 63741 ms
Total hot run time: 60482 ms

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 18, 2023

run buildall

1 similar comment
@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 18, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 18, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 710c3498895c3d10e8a2ead20a50e9a51c55fbb2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4726	4422	4509	4422
q2	365	150	158	150
q3	1465	1271	1199	1199
q4	1114	913	920	913
q5	3177	3158	3165	3158
q6	246	130	126	126
q7	995	486	489	486
q8	2240	2226	2209	2209
q9	6718	6658	6702	6658
q10	3230	3270	3274	3270
q11	319	196	201	196
q12	354	210	204	204
q13	4569	3814	3817	3814
q14	246	220	212	212
q15	568	532	518	518
q16	445	392	385	385
q17	1022	635	549	549
q18	7297	6927	6869	6869
q19	1552	1459	1383	1383
q20	507	314	303	303
q21	3112	2627	2622	2622
q22	355	281	287	281
Total cold run time: 44622 ms
Total hot run time: 39927 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4461	4384	4394	4384
q2	272	161	177	161
q3	3545	3528	3516	3516
q4	2400	2399	2382	2382
q5	5717	5731	5716	5716
q6	243	121	120	120
q7	2392	1861	1902	1861
q8	3533	3532	3530	3530
q9	9005	8987	8965	8965
q10	3940	3972	4002	3972
q11	512	376	382	376
q12	769	590	587	587
q13	4329	3569	3536	3536
q14	305	260	264	260
q15	584	524	526	524
q16	518	466	450	450
q17	1883	1859	1825	1825
q18	8668	8226	8353	8226
q19	1750	1752	1777	1752
q20	2256	1939	1928	1928
q21	6512	6169	6155	6155
q22	516	409	416	409
Total cold run time: 64110 ms
Total hot run time: 60635 ms

@doris-robot
Copy link

doris-robot commented Dec 18, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.6 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17216889645 Bytes

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Jan 4, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jan 4, 2024

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit eec7fac0a1b5eb35402d218fd5b0821d172a2c63, data reload: false

------ Round 1 ----------------------------------
q1	17647	5150	5146	5146
q2	2013	158	141	141
q3	10549	1107	1175	1107
q4	10182	791	816	791
q5	7766	3013	2984	2984
q6	210	140	140	140
q7	937	516	534	516
q8	9320	2022	2019	2019
q9	6866	6450	6390	6390
q10	8228	3067	3080	3067
q11	448	236	225	225
q12	389	234	230	230
q13	18025	3657	3658	3657
q14	241	212	202	202
q15	551	525	525	525
q16	453	394	401	394
q17	966	495	488	488
q18	7330	6780	6672	6672
q19	1589	1403	1417	1403
q20	770	344	343	343
q21	2837	2419	2444	2419
q22	377	325	327	325
Total cold run time: 107694 ms
Total hot run time: 39184 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5153	5140	5043	5043
q2	342	232	237	232
q3	3308	3295	3282	3282
q4	2126	2038	2054	2038
q5	5780	5808	5793	5793
q6	214	125	128	125
q7	2357	1900	1888	1888
q8	3383	3481	3471	3471
q9	8850	8783	8782	8782
q10	3778	3827	3867	3827
q11	599	474	505	474
q12	800	646	637	637
q13	7772	3236	3223	3223
q14	295	255	282	255
q15	582	512	520	512
q16	552	488	488	488
q17	1962	1798	1740	1740
q18	8679	8405	8274	8274
q19	1617	1589	1582	1582
q20	2169	1950	1942	1942
q21	5555	5273	5237	5237
q22	531	499	488	488
Total cold run time: 66404 ms
Total hot run time: 59333 ms

@doris-robot
Copy link

doris-robot commented Jan 4, 2024

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit eec7fac0a1b5eb35402d218fd5b0821d172a2c63, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	916	353	337	337
query2	6438	1935	1893	1893
query3	6653	220	208	208
query4	26308	22562	22494	22494
query5	4360	570	575	570
query6	270	188	174	174
query7	4581	277	267	267
query8	238	211	198	198
query9	8206	2603	2628	2603
query10	503	237	234	234
query11	16179	15695	15423	15423
query12	134	75	73	73
query13	1630	337	334	334
query14	11302	7044	7229	7044
query15	225	192	191	191
query16	6394	270	265	265
query17	1840	496	492	492
query18	1929	275	263	263
query19	195	136	131	131
query20	81	73	77	73
query21	187	99	100	99
query22	4902	4534	4432	4432
query23	32497	31427	31150	31150
query24	11965	2831	2804	2804
query25	609	342	337	337
query26	1700	143	145	143
query27	2920	283	277	277
query28	6926	1912	1897	1897
query29	1705	402	400	400
query30	281	148	145	145
query31	967	767	787	767
query32	95	60	53	53
query33	725	267	264	264
query34	873	464	469	464
query35	884	806	777	777
query36	1300	1212	1235	1212
query37	99	64	65	64
query38	3392	3298	3276	3276
query39	1325	1302	1285	1285
query40	303	95	89	89
query41	39	37	35	35
query42	90	84	83	83
query43	548	493	494	493
query44	1055	704	713	704
query45	192	193	189	189
query46	1064	639	658	639
query47	1647	1560	1577	1560
query48	327	263	265	263
query49	1167	317	338	317
query50	737	372	338	338
query51	5455	5336	5234	5234
query52	92	86	88	86
query53	219	148	147	147
query54	1373	574	570	570
query55	94	89	88	88
query56	204	193	189	189
query57	1001	927	956	927
query58	219	192	195	192
query59	2849	2646	2576	2576
query60	219	218	223	218
query61	89	87	90	87
query62	620	459	456	456
query63	170	141	147	141
query64	5920	1719	1772	1719
query65	3343	3259	3250	3250
query66	1327	346	335	335
query67	15647	14979	15193	14979
query68	12712	532	530	530
query69	522	256	249	249
query70	1804	1581	1519	1519
query71	491	215	228	215
query72	5703	3590	3563	3563
query73	2917	318	317	317
query74	6988	6445	6504	6445
query75	5363	2326	2275	2275
query76	6319	1066	1098	1066
query77	677	248	258	248
query78	9123	8601	8488	8488
query79	1035	523	515	515
query80	554	359	358	358
query81	462	209	207	207
query82	208	92	89	89
query83	163	138	138	138
query84	249	54	55	54
query85	952	291	277	277
query86	376	421	397	397
query87	3523	3427	3340	3340
query88	3037	2252	2264	2252
query89	350	258	253	253
query90	1873	194	195	194
query91	118	93	97	93
query92	61	50	52	50
query93	1423	438	500	438
query94	821	181	182	181
query95	464	413	425	413
query96	610	321	318	318
query97	4275	4144	4161	4144
query98	209	207	181	181
query99	1135	846	817	817
Total cold run time: 292261 ms
Total hot run time: 178641 ms

@doris-robot
Copy link

doris-robot commented Jan 4, 2024

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.68 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183819923 Bytes

@doris-robot
Copy link

doris-robot commented Jan 4, 2024

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit eec7fac0a1b5eb35402d218fd5b0821d172a2c63, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5494	5115	5128	5115
q2	386	153	155	153
q3	1470	1169	1153	1153
q4	1088	753	806	753
q5	3105	3123	3136	3123
q6	219	136	137	136
q7	963	558	509	509
q8	2151	2207	2200	2200
q9	6681	6687	6646	6646
q10	3184	3147	3124	3124
q11	350	233	219	219
q12	370	227	235	227
q13	4384	3594	3660	3594
q14	243	221	218	218
q15	574	539	519	519
q16	466	384	407	384
q17	1050	609	549	549
q18	7067	6755	6744	6744
q19	1639	1470	1389	1389
q20	593	354	336	336
q21	2910	2456	2444	2444
q22	373	324	309	309
Total cold run time: 44760 ms
Total hot run time: 39844 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5116	5165	5017	5017
q2	332	225	233	225
q3	3357	3339	3323	3323
q4	2145	2046	2054	2046
q5	5951	5919	5926	5919
q6	224	126	127	126
q7	2409	1915	1902	1902
q8	3545	3652	3660	3652
q9	9016	9007	8949	8949
q10	3865	3928	3889	3889
q11	578	499	479	479
q12	795	613	624	613
q13	3893	3180	3153	3153
q14	316	272	300	272
q15	597	528	532	528
q16	559	489	504	489
q17	2045	1816	1794	1794
q18	8738	8323	8293	8293
q19	1740	1711	1682	1682
q20	2269	1972	1970	1970
q21	5701	5384	5315	5315
q22	568	487	496	487
Total cold run time: 63759 ms
Total hot run time: 60123 ms

@github-actions
Copy link
Contributor

github-actions bot commented Jan 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 4, 2024

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 441fb49 into apache:master Jan 4, 2024
jackwener pushed a commit to jackwener/doris that referenced this pull request Jan 4, 2024
@BiteTheDDDDt @jackwener
[Bug](load) fix load failed on stream load tvf into agg state (apache…
06354fc 
…#28420)

fix load failed on stream load tvf into agg state
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@BiteTheDDDDt @HappenLee
[Bug](load) fix load failed on stream load tvf into agg state (apache…
a1356ad 
…#28420)

fix load failed on stream load tvf into agg state
@BiteTheDDDDt BiteTheDDDDt deleted the fix_1214_2 branch January 20, 2025 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zclllyybb zclllyybb zclllyybb approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@BiteTheDDDDt @doris-robot @hello-stephen @zclllyybb @zhangstar333 @dataroaring