[regression-test](Variant) add compaction case for variant and fix bugs by eldenmoon · Pull Request #28066 · apache/doris

eldenmoon · 2023-12-06T08:44:26Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

eldenmoon · 2023-12-06T08:44:34Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-12-06T08:52:11Z

be/src/vec/json/path_in_data.cpp

+    return pop_nfront(1);
+}
+
+PathInData PathInData::pop_nfront(size_t n) const {


warning: method 'pop_nfront' can be made static [readability-convert-member-functions-to-static]

be/src/vec/json/path_in_data.h:78:

- PathInData pop_nfront(size_t n) const; + static PathInData pop_nfront(size_t n) ;

Suggested change

PathInData PathInData::pop_nfront(size_t n) const {

PathInData PathInData::pop_nfront(size_t n) {

doris-robot · 2023-12-06T09:56:38Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.17 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17164771252 Bytes

doris-robot · 2023-12-06T09:58:47Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5f480f65fbc800f149e0fab5d10285ccc43bda2b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4723	4441	4437	4437
q2	367	160	153	153
q3	1474	1210	1245	1210
q4	1117	969	918	918
q5	3208	3207	3210	3207
q6	251	126	128	126
q7	990	504	482	482
q8	2252	2222	2219	2219
q9	6679	6703	6690	6690
q10	3214	3265	3285	3265
q11	324	212	208	208
q12	355	208	214	208
q13	4538	3809	3786	3786
q14	248	211	214	211
q15	567	524	518	518
q16	444	388	393	388
q17	1029	629	544	544
q18	7524	7254	7172	7172
q19	1528	1448	1397	1397
q20	569	347	331	331
q21	3121	2634	2651	2634
q22	359	288	302	288
Total cold run time: 44881 ms
Total hot run time: 40392 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4350	4359	4342	4342
q2	265	159	170	159
q3	3532	3516	3514	3514
q4	2386	2365	2375	2365
q5	5717	5736	5728	5728
q6	239	120	122	120
q7	2377	1903	1846	1846
q8	3524	3537	3538	3537
q9	9046	9021	9000	9000
q10	3919	3981	4003	3981
q11	507	388	389	388
q12	761	579	583	579
q13	4280	3585	3554	3554
q14	284	261	258	258
q15	566	524	514	514
q16	493	458	491	458
q17	1891	1854	1865	1854
q18	8597	8149	8049	8049
q19	1743	1743	1756	1743
q20	2250	1946	1945	1945
q21	6520	6202	6201	6201
q22	497	430	435	430
Total cold run time: 63744 ms
Total hot run time: 60565 ms

eldenmoon · 2023-12-06T10:20:08Z

run buildall

doris-robot · 2023-12-06T11:26:05Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.94 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17164468053 Bytes

doris-robot · 2023-12-06T13:19:35Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5969450d2e4d0295e862a3ca34e4e2122ce96e97, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4768	4485	4497	4485
q2	365	162	158	158
q3	1457	1274	1249	1249
q4	1106	925	926	925
q5	3201	3173	3197	3173
q6	248	131	129	129
q7	988	496	495	495
q8	2226	2225	2195	2195
q9	6707	6664	6686	6664
q10	3215	3251	3273	3251
q11	320	204	205	204
q12	354	216	208	208
q13	4569	3793	3816	3793
q14	245	208	220	208
q15	569	524	526	524
q16	450	387	395	387
q17	1016	598	586	586
q18	7405	7630	7135	7135
q19	1529	1402	1416	1402
q20	602	554	338	338
q21	3106	2623	2688	2623
q22	365	290	299	290
Total cold run time: 44811 ms
Total hot run time: 40422 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4423	4425	4396	4396
q2	267	162	178	162
q3	3541	3530	3530	3530
q4	2384	2374	2379	2374
q5	5752	5732	5756	5732
q6	242	121	121	121
q7	2383	1854	1862	1854
q8	3522	3520	3550	3520
q9	9028	9077	9027	9027
q10	3903	3971	3996	3971
q11	497	398	396	396
q12	777	600	590	590
q13	4307	3531	3551	3531
q14	288	252	266	252
q15	570	515	522	515
q16	484	458	468	458
q17	1890	1871	1878	1871
q18	8787	8440	8201	8201
q19	1740	1756	1763	1756
q20	2252	1959	1938	1938
q21	6533	6157	6149	6149
q22	494	426	412	412
Total cold run time: 64064 ms
Total hot run time: 60756 ms

xiaokang

LGTM

be/src/vec/json/path_in_data.cpp

be/src/vec/functions/function_cast.h

be/src/vec/json/path_in_data.cpp

be/src/olap/rowset/segment_v2/hierarchical_data_reader.cpp

eldenmoon · 2023-12-07T09:32:41Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-12-07T09:39:36Z

be/src/vec/json/path_in_data.cpp

+    return copy_pop_nfront(1);
+}
+
+PathInData PathInData::copy_pop_nfront(size_t n) const {


warning: method 'copy_pop_nfront' can be made static [readability-convert-member-functions-to-static]

be/src/vec/json/path_in_data.h:78:

- PathInData copy_pop_nfront(size_t n) const; + static PathInData copy_pop_nfront(size_t n) ;

Suggested change

PathInData PathInData::copy_pop_nfront(size_t n) const {

PathInData PathInData::copy_pop_nfront(size_t n) {

eldenmoon · 2023-12-07T09:42:34Z

run buildall

doris-robot · 2023-12-07T11:18:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.31 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17214615036 Bytes

doris-robot · 2023-12-07T11:19:58Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c6b475aae4afa92ac5e568c6a7cc15cc20f67922, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4688	4453	4460	4453
q2	366	149	159	149
q3	1458	1266	1205	1205
q4	1118	919	859	859
q5	3181	3186	3179	3179
q6	249	127	129	127
q7	984	506	493	493
q8	2208	2216	2210	2210
q9	6639	6714	6759	6714
q10	3187	3273	3275	3273
q11	327	211	203	203
q12	359	217	208	208
q13	4562	3994	3801	3801
q14	239	218	214	214
q15	563	524	510	510
q16	447	385	396	385
q17	1009	579	613	579
q18	7507	7437	7072	7072
q19	1521	1416	1440	1416
q20	559	355	316	316
q21	3123	2623	2701	2623
q22	366	299	301	299
Total cold run time: 44660 ms
Total hot run time: 40288 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4418	4421	4425	4421
q2	266	158	170	158
q3	3536	3530	3510	3510
q4	2396	2388	2376	2376
q5	5737	5723	5764	5723
q6	241	119	119	119
q7	2373	1868	1848	1848
q8	3529	3528	3557	3528
q9	9084	9070	8987	8987
q10	3931	3997	3987	3987
q11	494	354	377	354
q12	771	615	603	603
q13	4332	3574	3579	3574
q14	290	253	259	253
q15	587	509	522	509
q16	502	442	466	442
q17	1880	1847	1878	1847
q18	8616	8244	8324	8244
q19	1726	1765	1772	1765
q20	2257	1961	1938	1938
q21	6559	6191	6150	6150
q22	507	425	444	425
Total cold run time: 64032 ms
Total hot run time: 60761 ms

xiaokang

LGTM

github-actions · 2023-12-08T01:26:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-08T01:26:09Z

PR approved by anyone and no changes requested.

airborne12

LGTM

…gs (apache#28066)

github-actions bot reviewed Dec 6, 2023

View reviewed changes

xiaokang previously approved these changes Dec 7, 2023

View reviewed changes

be/src/vec/json/path_in_data.cpp Show resolved Hide resolved

be/src/vec/functions/function_cast.h Show resolved Hide resolved

be/src/vec/json/path_in_data.cpp Outdated Show resolved Hide resolved

be/src/olap/rowset/segment_v2/hierarchical_data_reader.cpp Show resolved Hide resolved

eldenmoon added 2 commits December 7, 2023 17:29

[regression-test](Variant) add compaction case for variant and fix bugs

158a31d

fix

d052145

eldenmoon dismissed xiaokang’s stale review via 855187f December 7, 2023 09:32

eldenmoon force-pushed the var-cases-compaction branch from 5969450 to 855187f Compare December 7, 2023 09:32

eldenmoon added the variant label Dec 7, 2023

github-actions bot reviewed Dec 7, 2023

View reviewed changes

rename pop_front to copy_pop_front

c6b475a

eldenmoon force-pushed the var-cases-compaction branch from 855187f to c6b475a Compare December 7, 2023 09:42

xiaokang approved these changes Dec 8, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2023

github-actions bot added the reviewed label Dec 8, 2023

airborne12 approved these changes Dec 8, 2023

View reviewed changes

eldenmoon merged commit 341822e into apache:master Dec 8, 2023

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[regression-test](Variant) add compaction case for variant and fix bu…

f8a46f7

…gs (apache#28066)

	PathInData PathInData::pop_nfront(size_t n) const {
	PathInData PathInData::pop_nfront(size_t n) {

	PathInData PathInData::copy_pop_nfront(size_t n) const {
	PathInData PathInData::copy_pop_nfront(size_t n) {

Conversation

eldenmoon commented Dec 6, 2023

Proposed changes

Further comments

Uh oh!

eldenmoon commented Dec 6, 2023

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

eldenmoon commented Dec 6, 2023

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Dec 7, 2023

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 7, 2023

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Dec 7, 2023

Uh oh!

doris-robot commented Dec 7, 2023

Uh oh!

doris-robot commented Dec 7, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 8, 2023

Uh oh!

github-actions bot commented Dec 8, 2023

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants