Skip to content

branch-3.1: [Fix](compaction) Remove incorrect DCHECK #54650 #54745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 19, 2025
Merged

branch-3.1: [Fix](compaction) Remove incorrect DCHECK #54650 #54745

morrySnow merged 1 commit into from
Aug 19, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 14, 2025

Cherry-picked from #54650

@bobhan1
[Fix](compaction) Remove incorrect DCHECK (#54650)
7b5f0ca 
### What problem does this PR solve?
This DCHECK asserts that there will be no new missed rows in rowid
conversion when converting delete bitmap marks for incremental rowsets.
That is, the delete bitmap marks should not be generated on merged rows
in input rowsets. However , this is not correct because new load may
generate duplicate delete bitmap marks if it only sees part of input
rowsets in its flush phase.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot requested a review from morrySnow as a code owner August 14, 2025 04:11
@Thearas
Copy link
Contributor

Thearas commented Aug 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 14, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-H: Total hot run time: 32982 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b5f0ca1285bbb2be6324ab49fd3d9d7994c307a, data reload: false

------ Round 1 ----------------------------------
q1	17597	5607	5395	5395
q2	2011	418	306	306
q3	11759	1273	767	767
q4	10260	890	466	466
q5	8624	2466	2234	2234
q6	184	164	135	135
q7	907	751	626	626
q8	9322	1489	1237	1237
q9	5339	4965	4934	4934
q10	6765	2270	1851	1851
q11	481	276	268	268
q12	337	358	205	205
q13	17767	3628	2987	2987
q14	225	228	218	218
q15	526	457	462	457
q16	415	422	388	388
q17	631	881	380	380
q18	7152	6484	6389	6389
q19	1217	983	550	550
q20	337	346	216	216
q21	3121	2223	1987	1987
q22	1031	1013	986	986
Total cold run time: 106008 ms
Total hot run time: 32982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5567	5536	5565	5536
q2	248	323	236	236
q3	2227	2632	2360	2360
q4	1333	1821	1394	1394
q5	4416	5011	5014	5011
q6	168	166	129	129
q7	2126	1991	1817	1817
q8	2653	2920	2744	2744
q9	7362	7327	7285	7285
q10	3039	3259	2715	2715
q11	579	510	495	495
q12	676	743	610	610
q13	3431	3788	3163	3163
q14	297	296	275	275
q15	521	473	460	460
q16	441	479	440	440
q17	1257	1776	1270	1270
q18	7618	7443	7419	7419
q19	793	1180	1091	1091
q20	2070	2050	1886	1886
q21	5478	5041	4729	4729
q22	1067	1086	1013	1013
Total cold run time: 53367 ms
Total hot run time: 52078 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-DS: Total hot run time: 192035 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b5f0ca1285bbb2be6324ab49fd3d9d7994c307a, data reload: false

query1	983	418	405	405
query2	6192	1991	1870	1870
query3	8688	199	193	193
query4	33698	24082	23523	23523
query5	3709	602	464	464
query6	289	201	187	187
query7	4214	509	322	322
query8	308	252	239	239
query9	9535	2592	2579	2579
query10	493	337	254	254
query11	18031	15422	15524	15422
query12	169	120	107	107
query13	1564	562	432	432
query14	9598	6521	6621	6521
query15	234	187	182	182
query16	8074	654	501	501
query17	1527	777	619	619
query18	2143	439	324	324
query19	208	189	166	166
query20	125	134	134	134
query21	213	127	113	113
query22	4553	4587	4343	4343
query23	35164	34494	34152	34152
query24	7676	2690	2731	2690
query25	523	492	434	434
query26	1172	297	171	171
query27	2725	487	349	349
query28	5730	2203	2199	2199
query29	789	594	475	475
query30	248	195	164	164
query31	1003	925	807	807
query32	89	61	56	56
query33	537	358	325	325
query34	769	852	523	523
query35	779	802	759	759
query36	1048	1102	992	992
query37	110	96	79	79
query38	3977	3983	4015	3983
query39	1552	1491	1480	1480
query40	204	117	106	106
query41	51	51	58	51
query42	136	106	105	105
query43	493	536	482	482
query44	1341	814	813	813
query45	185	182	179	179
query46	897	1067	672	672
query47	2055	2010	2002	2002
query48	402	415	350	350
query49	777	495	412	412
query50	681	688	435	435
query51	7307	7437	7201	7201
query52	99	101	92	92
query53	229	255	186	186
query54	565	537	483	483
query55	86	77	78	77
query56	271	276	258	258
query57	1274	1291	1198	1198
query58	247	222	232	222
query59	3028	3180	3001	3001
query60	285	286	257	257
query61	111	113	108	108
query62	787	757	692	692
query63	232	187	197	187
query64	4511	1027	636	636
query65	3547	3302	3276	3276
query66	977	409	339	339
query67	16346	15692	15815	15692
query68	7812	816	555	555
query69	489	310	265	265
query70	1212	1077	1088	1077
query71	387	297	257	257
query72	5733	3723	3880	3723
query73	637	738	356	356
query74	10465	9312	8988	8988
query75	3218	3146	2661	2661
query76	3069	1164	774	774
query77	529	359	274	274
query78	10306	10491	9645	9645
query79	3044	925	597	597
query80	624	537	447	447
query81	513	259	224	224
query82	564	122	87	87
query83	167	156	142	142
query84	260	99	88	88
query85	787	377	293	293
query86	394	306	265	265
query87	4372	4419	4201	4201
query88	5131	2394	2397	2394
query89	408	339	302	302
query90	1778	187	187	187
query91	134	157	108	108
query92	64	57	51	51
query93	1980	885	541	541
query94	689	405	270	270
query95	341	281	267	267
query96	500	618	282	282
query97	3248	3290	3149	3149
query98	227	208	203	203
query99	1495	1400	1312	1312
Total cold run time: 296140 ms
Total hot run time: 192035 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

ClickBench: Total hot run time: 28.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b5f0ca1285bbb2be6324ab49fd3d9d7994c307a, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.52	0.51	0.52
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.50	0.50
query10	0.55	0.53	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.63	0.61	0.60
query14	0.78	0.80	0.78
query15	0.84	0.83	0.81
query16	0.40	0.36	0.41
query17	1.00	1.02	0.96
query18	0.23	0.23	0.23
query19	1.84	1.79	1.73
query20	0.01	0.01	0.01
query21	15.42	0.93	0.56
query22	0.74	0.85	0.63
query23	15.11	1.46	0.56
query24	3.26	1.81	1.05
query25	0.21	0.05	0.06
query26	0.25	0.14	0.13
query27	0.06	0.04	0.05
query28	13.27	0.99	0.44
query29	12.58	3.91	3.24
query30	0.24	0.09	0.07
query31	2.82	0.60	0.38
query32	3.23	0.54	0.46
query33	3.01	2.99	3.05
query34	16.64	5.22	4.59
query35	4.63	4.60	4.62
query36	0.64	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.39 s
Total hot run time: 28.78 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.44% (12707/27965)
Line Coverage 36.32% (113199/311707)
Region Coverage 33.96% (64798/190825)
Branch Coverage 30.99% (33998/109708)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.51% (20797/27541)
Line Coverage 68.76% (213962/311192)
Region Coverage 67.35% (129118/191714)
Branch Coverage 60.97% (67232/110268)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.51% (20797/27541)
Line Coverage 68.76% (213962/311192)
Region Coverage 67.35% (129118/191714)
Branch Coverage 60.97% (67232/110268)

@morrySnow morrySnow merged commit 71e1a18 into branch-3.1 Aug 19, 2025
22 of 23 checks passed
@morrySnow morrySnow deleted the auto-pick-54650-branch-3.1 branch August 19, 2025 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @bobhan1