Skip to content

branch-3.1: [Fix](mow) Should use rowset's schema when calculate delete bitmaps between segments #54351 #54366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 6, 2025
Merged

branch-3.1: [Fix](mow) Should use rowset's schema when calculate delete bitmaps between segments #54351 #54366

morrySnow merged 1 commit into from
Aug 6, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Aug 6, 2025

pick #54351

@bobhan1
[Fix](mow) Should use rowset's schema when calculate delete bitmaps b…
97a71ab 
…etween segments (apache#54351)

Currently, doris use tablet meta's schema to judge if the rowset has
sequence column. This may be not correct if user use
`alter table ${tbl} ENABLE FEATURE "SEQUENCE_LOAD" WITH PROPERTIES
("function_column.sequence_type"="xxx");` statement to add sequence
column for table and will cause data correctness problem.
@bobhan1 bobhan1 requested a review from morrySnow as a code owner August 6, 2025 03:00
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

TPC-H: Total hot run time: 32541 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97a71ab038b7b6aef1e38bc9c81b1c712bdbd19a, data reload: false

------ Round 1 ----------------------------------
q1	17583	5521	5403	5403
q2	2068	281	159	159
q3	10500	1241	736	736
q4	10206	879	457	457
q5	7685	2405	2145	2145
q6	183	162	132	132
q7	888	745	620	620
q8	9354	1439	1129	1129
q9	5171	4890	4956	4890
q10	6764	2289	1832	1832
q11	478	283	268	268
q12	341	350	215	215
q13	17820	3608	3013	3013
q14	225	226	209	209
q15	526	470	474	470
q16	415	427	392	392
q17	596	859	349	349
q18	7127	6497	6446	6446
q19	1605	965	554	554
q20	325	352	199	199
q21	2808	2182	1939	1939
q22	1051	1055	984	984
Total cold run time: 103719 ms
Total hot run time: 32541 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5674	5530	5541	5530
q2	242	334	241	241
q3	2280	2600	2314	2314
q4	1394	1828	1365	1365
q5	4463	4878	4990	4878
q6	167	167	125	125
q7	2089	1978	1847	1847
q8	2647	2871	2703	2703
q9	7313	7267	7304	7267
q10	2996	3341	2765	2765
q11	577	504	487	487
q12	655	731	622	622
q13	3430	3832	3146	3146
q14	294	297	280	280
q15	532	477	476	476
q16	459	494	423	423
q17	1236	1738	1281	1281
q18	7656	7626	7256	7256
q19	839	900	1142	900
q20	2042	2077	1903	1903
q21	5402	4937	4563	4563
q22	1081	1060	1006	1006
Total cold run time: 53468 ms
Total hot run time: 51378 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

TPC-DS: Total hot run time: 196961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97a71ab038b7b6aef1e38bc9c81b1c712bdbd19a, data reload: false

query1	1309	918	904	904
query2	6238	1903	1870	1870
query3	10956	4469	4420	4420
query4	33211	23706	23829	23706
query5	4420	610	459	459
query6	294	215	185	185
query7	4015	493	318	318
query8	320	253	237	237
query9	9162	2604	2589	2589
query10	479	319	248	248
query11	17965	15384	15217	15217
query12	159	109	104	104
query13	1542	529	405	405
query14	9266	6639	6911	6639
query15	227	212	189	189
query16	7813	694	514	514
query17	1592	769	615	615
query18	2016	432	316	316
query19	204	193	175	175
query20	134	128	123	123
query21	213	138	115	115
query22	4572	4739	4466	4466
query23	35210	34286	33969	33969
query24	7786	2691	2665	2665
query25	480	468	421	421
query26	1204	284	168	168
query27	2776	455	348	348
query28	5697	2263	2191	2191
query29	671	576	453	453
query30	234	196	172	172
query31	998	907	890	890
query32	70	59	54	54
query33	511	361	296	296
query34	740	865	525	525
query35	810	818	739	739
query36	1002	1049	965	965
query37	108	95	66	66
query38	4067	4075	3997	3997
query39	1499	1467	1494	1467
query40	205	119	104	104
query41	50	50	52	50
query42	131	110	104	104
query43	528	532	502	502
query44	1353	841	831	831
query45	184	186	172	172
query46	892	1068	677	677
query47	1961	1956	1914	1914
query48	423	433	365	365
query49	796	518	409	409
query50	675	693	440	440
query51	7455	7447	7205	7205
query52	105	104	93	93
query53	234	258	194	194
query54	547	545	485	485
query55	84	79	80	79
query56	267	264	263	263
query57	1302	1260	1249	1249
query58	263	249	232	232
query59	3144	3206	3025	3025
query60	302	313	290	290
query61	141	144	138	138
query62	845	764	709	709
query63	261	198	195	195
query64	4375	1114	737	737
query65	3407	3281	3209	3209
query66	962	429	320	320
query67	16112	15963	15501	15501
query68	4686	854	573	573
query69	488	324	278	278
query70	1237	1165	1099	1099
query71	394	295	266	266
query72	5943	3790	3822	3790
query73	640	735	356	356
query74	10078	9277	9033	9033
query75	3171	3182	2663	2663
query76	3131	1166	769	769
query77	518	368	280	280
query78	10385	10357	9626	9626
query79	3177	927	595	595
query80	745	523	428	428
query81	494	261	221	221
query82	903	128	89	89
query83	175	163	143	143
query84	250	101	80	80
query85	792	391	306	306
query86	385	319	298	298
query87	4348	4421	4246	4246
query88	5111	2413	2400	2400
query89	426	337	294	294
query90	1805	190	191	190
query91	142	143	110	110
query92	70	55	54	54
query93	2685	885	531	531
query94	676	415	308	308
query95	341	277	273	273
query96	490	612	289	289
query97	3243	3240	3177	3177
query98	225	211	210	210
query99	1570	1409	1294	1294
Total cold run time: 295698 ms
Total hot run time: 196961 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

ClickBench: Total hot run time: 28.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97a71ab038b7b6aef1e38bc9c81b1c712bdbd19a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.11	0.11
query5	0.54	0.53	0.54
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.10
query13	0.62	0.60	0.60
query14	0.76	0.84	0.77
query15	0.85	0.83	0.82
query16	0.37	0.40	0.39
query17	1.06	0.98	1.03
query18	0.24	0.22	0.22
query19	1.83	1.86	1.81
query20	0.02	0.01	0.01
query21	15.38	0.95	0.62
query22	0.73	0.93	0.58
query23	15.08	1.38	0.51
query24	2.96	1.18	0.65
query25	0.25	0.11	0.05
query26	0.25	0.14	0.14
query27	0.06	0.05	0.04
query28	13.64	1.03	0.43
query29	12.56	4.00	3.32
query30	0.24	0.10	0.07
query31	2.82	0.60	0.37
query32	3.22	0.54	0.46
query33	2.97	3.00	3.02
query34	16.58	5.18	4.51
query35	4.51	4.54	4.51
query36	0.63	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.22 s
Total hot run time: 28.42 s

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.40% (12641/27842)
Line Coverage 36.25% (112659/310776)
Region Coverage 35.30% (58233/164980)
Branch Coverage 32.48% (31661/97464)

@morrySnow morrySnow changed the title branch-3.1: [Fix](mow) Should use rowset's schema when calculate delete bitmaps between segments (#54351) branch-3.1: [Fix](mow) Should use rowset's schema when calculate delete bitmaps between segments #54351 Aug 6, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 6, 2025

BE Regression && UT Coverage Report

Increment line coverage 36.84% (7/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.19% (20959/27510)
Line Coverage 69.58% (216097/310569)
Region Coverage 67.58% (129301/191321)
Branch Coverage 61.16% (67272/109994)

@morrySnow morrySnow merged commit 6fdcc40 into apache:branch-3.1 Aug 6, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1 @hello-stephen @doris-robot @morrySnow