Skip to content

branch-3.1: [Opt](cloud-mow) BE can limit the size of get_delete_bitmap response roughly #55712 #56532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Sep 28, 2025
Merged

branch-3.1: [Opt](cloud-mow) BE can limit the size of get_delete_bitmap response roughly #55712 #56532

morrySnow merged 2 commits into from
Sep 28, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Sep 28, 2025 •
edited by morrySnow
Loading

picked from #55712

@bobhan1
[Opt](cloud-mow) BE can limit the size of get_delete_bitmap respons…
dc88448 
…e roughly (apache#55712)

Add BE conf `get_delete_bitmap_bytes_threshold` to limit the delete
bitmap size in `get_delete_bitmap` reponse. The MS will return the
current results to BE immediately when the size of delete bitmap fetched
from fdb in MS reached this theshold the first time, and BE will make
subsequent RPCs to get the remaining rowsets' results.

This feature is used to reduce the pressure of MS when there are large
delete bitmaps stored in fdb.
@bobhan1 bobhan1 requested a review from morrySnow as a code owner September 28, 2025 02:15
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 force-pushed the branch-3.1-pick-55712 branch from 467b50d to ae5ece7 Compare September 28, 2025 02:19
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 28, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 28, 2025

Cloud UT Coverage Report

Increment line coverage 31.25% (5/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.33% (1235/1500)
Line Coverage 66.23% (22270/33627)
Region Coverage 67.64% (11161/16500)
Branch Coverage 57.26% (5901/10306)

@bobhan1 bobhan1 force-pushed the branch-3.1-pick-55712 branch from ae5ece7 to ab5c36a Compare September 28, 2025 03:16
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

Cloud UT Coverage Report

Increment line coverage 31.25% (5/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.33% (1235/1500)
Line Coverage 66.17% (22251/33627)
Region Coverage 67.63% (11159/16500)
Branch Coverage 57.19% (5894/10306)

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

TPC-H: Total hot run time: 33064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab5c36a10c2d71fe48bd13f145e8c24e1ec311c6, data reload: false

------ Round 1 ----------------------------------
q1	17815	5488	5531	5488
q2	2044	407	293	293
q3	12025	1245	773	773
q4	10246	892	459	459
q5	8555	2443	2180	2180
q6	190	166	133	133
q7	929	763	620	620
q8	9357	1459	1183	1183
q9	5331	5030	4938	4938
q10	6807	2281	1832	1832
q11	471	283	262	262
q12	339	371	216	216
q13	17786	3634	3027	3027
q14	228	219	211	211
q15	540	467	462	462
q16	426	422	380	380
q17	597	872	354	354
q18	6978	6494	6419	6419
q19	1360	972	567	567
q20	352	357	205	205
q21	2948	2263	2079	2079
q22	1071	1021	983	983
Total cold run time: 106395 ms
Total hot run time: 33064 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5531	5513	5496	5496
q2	238	320	242	242
q3	2264	2654	2307	2307
q4	1403	1864	1373	1373
q5	4463	5026	5022	5022
q6	172	166	132	132
q7	2097	1996	1852	1852
q8	2639	2878	2733	2733
q9	7309	7275	7239	7239
q10	3060	3240	2771	2771
q11	589	508	500	500
q12	701	777	635	635
q13	3465	3831	3173	3173
q14	295	307	291	291
q15	515	468	462	462
q16	423	479	433	433
q17	1251	1749	1258	1258
q18	7602	7556	7292	7292
q19	882	1146	1145	1145
q20	1999	2046	1895	1895
q21	5451	4859	4640	4640
q22	1103	1063	1015	1015
Total cold run time: 53452 ms
Total hot run time: 51906 ms

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

TPC-DS: Total hot run time: 192802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ab5c36a10c2d71fe48bd13f145e8c24e1ec311c6, data reload: false

query1	950	398	381	381
query2	6230	2059	1959	1959
query3	8685	197	194	194
query4	33132	23595	24063	23595
query5	4620	594	455	455
query6	296	193	184	184
query7	4224	501	321	321
query8	316	259	243	243
query9	9422	2602	2594	2594
query10	502	341	267	267
query11	18237	15448	15157	15157
query12	161	109	107	107
query13	1566	551	430	430
query14	9595	6681	6687	6681
query15	234	188	175	175
query16	8044	665	544	544
query17	1499	761	598	598
query18	2108	427	327	327
query19	217	190	166	166
query20	127	124	123	123
query21	214	132	121	121
query22	4696	4571	4441	4441
query23	34983	34541	34100	34100
query24	7519	2709	2753	2709
query25	561	497	415	415
query26	1060	298	181	181
query27	2770	480	368	368
query28	5714	2253	2227	2227
query29	782	597	446	446
query30	256	204	170	170
query31	1010	921	804	804
query32	88	65	56	56
query33	488	376	318	318
query34	741	835	539	539
query35	803	803	726	726
query36	1023	1073	960	960
query37	110	90	68	68
query38	4127	4055	3974	3974
query39	1572	1477	1511	1477
query40	205	118	103	103
query41	49	48	48	48
query42	125	111	107	107
query43	525	537	497	497
query44	1337	833	844	833
query45	189	181	176	176
query46	902	1054	687	687
query47	1972	1980	1920	1920
query48	422	437	345	345
query49	762	486	412	412
query50	726	713	441	441
query51	7247	7282	7312	7282
query52	106	108	94	94
query53	242	271	203	203
query54	564	552	480	480
query55	85	79	78	78
query56	279	283	262	262
query57	1271	1295	1206	1206
query58	244	218	216	216
query59	3204	3275	3084	3084
query60	294	303	278	278
query61	122	115	118	115
query62	814	762	686	686
query63	243	195	194	194
query64	4490	1022	636	636
query65	3343	3291	3297	3291
query66	907	407	315	315
query67	16184	15938	15557	15557
query68	6413	838	561	561
query69	492	310	281	281
query70	1156	1165	1144	1144
query71	362	295	268	268
query72	5811	3822	3867	3822
query73	646	748	364	364
query74	10157	9448	9302	9302
query75	3261	3167	2718	2718
query76	2973	1185	779	779
query77	489	380	292	292
query78	10323	10550	9583	9583
query79	3600	837	600	600
query80	598	544	443	443
query81	498	253	222	222
query82	571	115	90	90
query83	161	163	153	153
query84	240	91	78	78
query85	786	353	315	315
query86	388	320	307	307
query87	4334	4410	4226	4226
query88	5210	2405	2487	2405
query89	396	331	290	290
query90	1748	194	206	194
query91	138	143	115	115
query92	63	55	52	52
query93	1877	935	551	551
query94	695	419	269	269
query95	336	275	274	274
query96	487	605	280	280
query97	3200	3258	3162	3162
query98	222	212	199	199
query99	1348	1404	1316	1316
Total cold run time: 294514 ms
Total hot run time: 192802 ms

@doris-robot
Copy link

doris-robot commented Sep 28, 2025

ClickBench: Total hot run time: 29.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ab5c36a10c2d71fe48bd13f145e8c24e1ec311c6, data reload: false

query1	0.03	0.04	0.03
query2	0.09	0.04	0.03
query3	0.23	0.06	0.06
query4	1.64	0.09	0.10
query5	0.51	0.51	0.51
query6	1.12	0.74	0.75
query7	0.03	0.02	0.02
query8	0.06	0.04	0.05
query9	0.58	0.51	0.50
query10	0.56	0.56	0.55
query11	0.17	0.12	0.12
query12	0.16	0.12	0.13
query13	0.62	0.59	0.60
query14	0.77	0.81	0.79
query15	0.87	0.84	0.84
query16	0.37	0.37	0.38
query17	1.07	1.05	1.08
query18	0.20	0.20	0.20
query19	1.98	1.92	1.92
query20	0.02	0.01	0.01
query21	15.38	0.94	0.64
query22	0.76	0.77	0.68
query23	14.81	1.49	0.67
query24	2.19	0.36	0.22
query25	0.15	0.10	0.08
query26	0.28	0.18	0.18
query27	0.08	0.09	0.08
query28	13.54	1.25	0.57
query29	12.66	4.02	3.40
query30	0.25	0.09	0.06
query31	2.82	0.59	0.41
query32	3.22	0.57	0.50
query33	3.02	3.08	3.07
query34	16.48	5.22	4.54
query35	4.60	4.68	4.60
query36	0.63	0.49	0.48
query37	0.21	0.17	0.16
query38	0.16	0.15	0.16
query39	0.06	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.06
query42	0.07	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 102.77 s
Total hot run time: 29.38 s

@morrySnow morrySnow merged commit dccfc86 into apache:branch-3.1 Sep 28, 2025
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1 @hello-stephen @doris-robot @morrySnow