Skip to content

[fix](compaction) fix the issue of writing segment with 0 rows during compaction #38251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Jul 24, 2024
Merged

[fix](compaction) fix the issue of writing segment with 0 rows during compaction #38251

dataroaring merged 3 commits into from
Jul 24, 2024

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Jul 23, 2024 •
edited
Loading

#37960 introduced this issue

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

dataroaring
dataroaring previously approved these changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

dataroaring commented Jul 23, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
@luwei16
Copy link
Contributor Author

luwei16 commented Jul 23, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

TPC-H: Total hot run time: 39951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e2bb50b15b164dc31c39fbc5fd91f4f0d2e0d946, data reload: false

------ Round 1 ----------------------------------
q1	17629	5047	4248	4248
q2	2019	190	184	184
q3	10521	1212	1162	1162
q4	10178	815	851	815
q5	7588	2671	2684	2671
q6	221	141	139	139
q7	956	592	593	592
q8	9218	2048	2071	2048
q9	8676	6539	6571	6539
q10	8827	3797	3799	3797
q11	454	238	238	238
q12	408	226	229	226
q13	17861	2969	3001	2969
q14	292	234	231	231
q15	529	485	511	485
q16	499	377	379	377
q17	966	690	718	690
q18	7976	7643	7390	7390
q19	7725	1497	1322	1322
q20	658	329	318	318
q21	4916	3226	3337	3226
q22	357	284	290	284
Total cold run time: 118474 ms
Total hot run time: 39951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4257	4251	4251
q2	358	282	276	276
q3	3134	2919	2973	2919
q4	1948	1750	1779	1750
q5	5755	5764	5651	5651
q6	237	148	141	141
q7	2370	1985	1951	1951
q8	3384	3425	3479	3425
q9	9052	9226	8930	8930
q10	4135	3806	3882	3806
q11	611	495	500	495
q12	829	666	653	653
q13	16506	3207	3241	3207
q14	331	301	291	291
q15	519	491	504	491
q16	481	446	440	440
q17	1820	1543	1519	1519
q18	8230	7896	7830	7830
q19	2875	1613	1478	1478
q20	2157	1881	1845	1845
q21	14666	4778	4696	4696
q22	585	498	508	498
Total cold run time: 84400 ms
Total hot run time: 56543 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

TPC-DS: Total hot run time: 174493 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e2bb50b15b164dc31c39fbc5fd91f4f0d2e0d946, data reload: false

query1	904	378	381	378
query2	6461	1995	1899	1899
query3	6656	204	217	204
query4	22977	17272	17415	17272
query5	3594	499	495	495
query6	282	177	165	165
query7	4595	285	289	285
query8	243	196	191	191
query9	8405	2432	2425	2425
query10	436	297	278	278
query11	11934	9911	10072	9911
query12	123	90	87	87
query13	1651	389	367	367
query14	10579	7805	8079	7805
query15	235	167	171	167
query16	7615	504	444	444
query17	1592	581	553	553
query18	1773	292	287	287
query19	201	156	155	155
query20	91	88	88	88
query21	207	132	126	126
query22	4279	4002	3956	3956
query23	34053	34115	33748	33748
query24	11125	2943	2954	2943
query25	661	473	404	404
query26	1221	155	156	155
query27	2335	280	284	280
query28	6905	2087	2094	2087
query29	929	694	655	655
query30	247	151	155	151
query31	998	748	753	748
query32	104	53	53	53
query33	757	347	346	346
query34	918	493	517	493
query35	879	763	783	763
query36	1178	1014	1001	1001
query37	152	82	85	82
query38	2942	2900	2846	2846
query39	935	840	868	840
query40	218	131	134	131
query41	49	49	46	46
query42	127	105	108	105
query43	520	476	474	474
query44	1255	738	735	735
query45	196	159	163	159
query46	1088	770	744	744
query47	1821	1776	1764	1764
query48	380	298	292	292
query49	868	423	432	423
query50	791	395	391	391
query51	6835	6615	6706	6615
query52	113	96	95	95
query53	366	297	296	296
query54	932	459	451	451
query55	74	74	76	74
query56	302	282	283	282
query57	1127	1036	1019	1019
query58	248	259	268	259
query59	2943	2653	2719	2653
query60	319	293	287	287
query61	101	97	123	97
query62	797	659	646	646
query63	335	297	299	297
query64	9428	2281	1683	1683
query65	3168	3116	3059	3059
query66	707	330	328	328
query67	15360	14862	14950	14862
query68	7324	569	566	566
query69	682	437	395	395
query70	1245	1129	1091	1091
query71	496	298	279	279
query72	8267	5461	5712	5461
query73	820	332	329	329
query74	6190	5743	5631	5631
query75	4296	2660	2695	2660
query76	4564	961	901	901
query77	717	325	324	324
query78	10939	10415	9092	9092
query79	6593	539	538	538
query80	1537	503	498	498
query81	566	226	226	226
query82	634	147	134	134
query83	197	172	169	169
query84	277	91	97	91
query85	1388	329	304	304
query86	446	332	330	330
query87	3305	3102	3093	3093
query88	4503	2415	2388	2388
query89	499	392	396	392
query90	1871	196	196	196
query91	131	103	104	103
query92	62	50	54	50
query93	4304	516	522	516
query94	1068	268	287	268
query95	408	365	323	323
query96	642	271	269	269
query97	3205	3050	3005	3005
query98	223	203	196	196
query99	1656	1289	1275	1275
Total cold run time: 290695 ms
Total hot run time: 174493 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e2bb50b15b164dc31c39fbc5fd91f4f0d2e0d946, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.50	0.50	0.49
query6	1.13	0.72	0.73
query7	0.03	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.48	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.76	0.78	0.77
query15	0.86	0.80	0.83
query16	0.36	0.35	0.36
query17	1.00	1.01	1.00
query18	0.22	0.22	0.22
query19	1.78	1.80	1.67
query20	0.02	0.01	0.01
query21	15.43	0.80	0.67
query22	4.59	6.86	2.16
query23	18.30	1.35	1.25
query24	2.08	0.23	0.22
query25	0.14	0.08	0.09
query26	0.29	0.21	0.22
query27	0.46	0.23	0.23
query28	13.31	1.02	1.00
query29	12.62	3.33	3.27
query30	0.26	0.06	0.06
query31	2.85	0.39	0.39
query32	3.29	0.48	0.47
query33	2.89	2.92	2.93
query34	17.15	4.35	4.45
query35	4.41	4.42	4.39
query36	0.65	0.48	0.46
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 110.21 s
Total hot run time: 30.82 s

dataroaring
dataroaring approved these changes Jul 24, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 24, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 24, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 83276c9 into apache:master Jul 24, 2024
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@luwei16 @dataroaring
[fix](compaction) fix the issue of writing segment with 0 rows during…
b5c4643 
… compaction (#38251)

#37960 introduced this issue
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Aug 4, 2024
@luwei16
[fix](compaction) fix the issue of writing segment with 0 rows during…
75699c6 
… compaction (apache#38251)

apache#37960 introduced this issue
@luwei16 luwei16 mentioned this pull request Aug 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 5, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
0f69a2a 
…ows during compaction (#37960)(#38251)(#38356) (#38835)

pick master #37960 #38251 #38356
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Aug 13, 2024
@luwei16
[fix](compaction) fix the issue of writing segment with 0 rows during…
91c30d5 
… compaction (apache#38251)

apache#37960 introduced this issue
@luwei16 luwei16 mentioned this pull request Aug 13, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 14, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
e217aab 
…ows during compaction (#37960) (#38251) (#38356)  (#39263)

pick master #37960 #38251 #38356
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
2a08d5a 
…ows during compaction (apache#37960) (apache#38251) (apache#38356)  (apache#39263)

pick master apache#37960 apache#38251 apache#38356
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@luwei16 @doris-robot @dataroaring @Yukang-Lian