Skip to content

Conversation

@qzsee
Copy link
Contributor

@qzsee qzsee commented Sep 12, 2025

What problem does this PR solve?

Make the read slice size variable configurable to reduce the impact caused by read amplification.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor

run buildall

@qzsee qzsee force-pushed the add-read-slice-config branch from 78aac9b to 84235f2 Compare September 12, 2025 06:50
@kaka11chen
Copy link
Contributor

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 84.84% (1522/1794)
Line Coverage 68.17% (27182/39876)
Region Coverage 68.97% (13461/19516)
Branch Coverage 59.11% (7192/12168)

@dqz123
Copy link

dqz123 commented Sep 12, 2025

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 84.84% (1522/1794)
Line Coverage 68.16% (27178/39876)
Region Coverage 68.94% (13454/19516)
Branch Coverage 59.16% (7198/12168)

@doris-robot
Copy link

TPC-H: Total hot run time: 35261 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8646d2d54ecbc9011fd65f954ad40fc274c5247d, data reload: false

------ Round 1 ----------------------------------
q1	16937	5265	5115	5115
q2	2007	317	217	217
q3	9982	1345	785	785
q4	10209	1032	563	563
q5	7619	2430	2400	2400
q6	184	169	137	137
q7	971	804	653	653
q8	9355	1274	1182	1182
q9	6997	5118	5179	5118
q10	6945	2400	2014	2014
q11	502	313	294	294
q12	361	375	236	236
q13	17774	3700	3055	3055
q14	234	241	227	227
q15	579	502	494	494
q16	1004	1013	969	969
q17	641	883	371	371
q18	8042	7216	7263	7216
q19	1190	952	566	566
q20	353	343	230	230
q21	3801	3307	2444	2444
q22	1114	1056	975	975
Total cold run time: 106801 ms
Total hot run time: 35261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5268	5157	5169	5157
q2	249	339	234	234
q3	2167	2712	2326	2326
q4	1431	1773	1286	1286
q5	4326	4584	4502	4502
q6	222	174	136	136
q7	2075	2000	1786	1786
q8	2756	2735	2667	2667
q9	7315	7291	7339	7291
q10	3112	3321	2876	2876
q11	586	526	514	514
q12	684	820	693	693
q13	3596	4132	3264	3264
q14	270	291	268	268
q15	522	513	481	481
q16	1095	1116	1170	1116
q17	1209	1629	1371	1371
q18	8024	7612	7681	7612
q19	810	821	866	821
q20	2029	1961	1874	1874
q21	4886	4275	4360	4275
q22	1092	1059	993	993
Total cold run time: 53724 ms
Total hot run time: 51543 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189327 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8646d2d54ecbc9011fd65f954ad40fc274c5247d, data reload: false

query1	1056	459	449	449
query2	6582	1740	1683	1683
query3	6756	223	229	223
query4	26669	23659	22941	22941
query5	4407	659	484	484
query6	344	255	234	234
query7	4667	517	295	295
query8	305	271	244	244
query9	8657	2652	2631	2631
query10	488	331	275	275
query11	15799	15097	15227	15097
query12	174	120	110	110
query13	1666	587	440	440
query14	11003	9516	9483	9483
query15	224	202	180	180
query16	7273	682	508	508
query17	1210	788	627	627
query18	1999	423	335	335
query19	211	196	173	173
query20	132	128	129	128
query21	214	130	118	118
query22	4015	4110	3938	3938
query23	34011	32999	32865	32865
query24	8352	2423	2446	2423
query25	602	508	436	436
query26	1238	280	164	164
query27	2746	519	357	357
query28	4259	2264	2216	2216
query29	874	644	515	515
query30	303	229	203	203
query31	930	827	725	725
query32	83	75	71	71
query33	592	399	330	330
query34	814	858	537	537
query35	842	820	747	747
query36	943	1040	943	943
query37	122	117	86	86
query38	3518	3544	3562	3544
query39	1494	1410	1470	1410
query40	220	137	123	123
query41	67	62	65	62
query42	133	122	122	122
query43	530	525	478	478
query44	1384	857	851	851
query45	186	178	171	171
query46	866	1028	650	650
query47	1737	1785	1716	1716
query48	415	439	311	311
query49	764	513	417	417
query50	687	688	408	408
query51	3871	4051	3914	3914
query52	117	116	99	99
query53	255	273	192	192
query54	604	594	534	534
query55	89	85	84	84
query56	327	341	304	304
query57	1176	1170	1105	1105
query58	276	262	273	262
query59	2623	2643	2574	2574
query60	366	357	343	343
query61	212	187	189	187
query62	811	731	684	684
query63	237	201	204	201
query64	4589	1261	920	920
query65	4068	4012	4064	4012
query66	1138	436	338	338
query67	15586	15313	15154	15154
query68	8336	935	580	580
query69	494	327	281	281
query70	1386	1287	1319	1287
query71	583	343	317	317
query72	5910	5027	5086	5027
query73	757	651	361	361
query74	8914	9226	9025	9025
query75	3981	3320	2879	2879
query76	3762	1175	760	760
query77	795	408	314	314
query78	9689	9775	8953	8953
query79	2125	794	585	585
query80	685	565	548	548
query81	501	262	231	231
query82	409	157	136	136
query83	310	259	252	252
query84	305	115	91	91
query85	879	453	515	453
query86	382	310	324	310
query87	3795	3776	3661	3661
query88	3543	2204	2207	2204
query89	388	328	299	299
query90	1919	223	220	220
query91	168	161	214	161
query92	86	68	66	66
query93	1684	1017	641	641
query94	679	445	340	340
query95	403	322	313	313
query96	482	572	275	275
query97	2940	2982	2876	2876
query98	245	215	214	214
query99	1436	1444	1283	1283
Total cold run time: 276879 ms
Total hot run time: 189327 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8646d2d54ecbc9011fd65f954ad40fc274c5247d, data reload: false

query1	0.05	0.05	0.05
query2	0.12	0.06	0.06
query3	0.30	0.07	0.08
query4	1.62	0.09	0.09
query5	0.26	0.25	0.24
query6	1.16	0.65	0.65
query7	0.03	0.03	0.02
query8	0.07	0.05	0.05
query9	0.66	0.54	0.53
query10	0.57	0.59	0.58
query11	0.25	0.12	0.12
query12	0.25	0.14	0.13
query13	0.65	0.63	0.64
query14	1.05	1.07	1.07
query15	0.94	0.88	0.87
query16	0.40	0.41	0.40
query17	1.07	1.07	1.05
query18	0.23	0.21	0.22
query19	1.97	1.86	1.84
query20	0.02	0.02	0.01
query21	15.39	1.00	0.70
query22	0.93	1.04	0.85
query23	14.69	1.60	0.78
query24	5.41	0.65	0.34
query25	0.17	0.09	0.08
query26	0.54	0.21	0.17
query27	0.10	0.09	0.09
query28	11.02	1.20	0.58
query29	12.54	3.97	3.38
query30	0.32	0.12	0.10
query31	2.84	0.63	0.44
query32	3.25	0.61	0.51
query33	3.07	3.17	3.23
query34	16.88	5.58	4.78
query35	4.85	4.92	4.92
query36	0.65	0.54	0.52
query37	0.24	0.19	0.19
query38	0.18	0.18	0.17
query39	0.06	0.05	0.05
query40	0.21	0.18	0.18
query41	0.11	0.05	0.06
query42	0.07	0.06	0.06
query43	0.06	0.06	0.05
Total cold run time: 105.25 s
Total hot run time: 30.83 s

@dqz123
Copy link

dqz123 commented Sep 12, 2025

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 84.84% (1522/1794)
Line Coverage 68.16% (27179/39876)
Region Coverage 68.98% (13462/19516)
Branch Coverage 59.14% (7196/12168)

@doris-robot
Copy link

TPC-H: Total hot run time: 34788 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d42c4bd01a234fdf2fc35f30308f607cabcc45e7, data reload: false

------ Round 1 ----------------------------------
q1	16540	5287	5054	5054
q2	2035	319	244	244
q3	9755	1308	723	723
q4	9885	1028	508	508
q5	7525	2488	2369	2369
q6	194	177	143	143
q7	951	777	638	638
q8	9261	1373	1133	1133
q9	6980	5120	5209	5120
q10	6916	2400	1995	1995
q11	482	316	288	288
q12	351	371	244	244
q13	17769	3696	3022	3022
q14	242	252	224	224
q15	565	511	500	500
q16	1008	992	962	962
q17	622	868	364	364
q18	7552	7283	7064	7064
q19	1244	958	560	560
q20	352	343	238	238
q21	3769	3165	2379	2379
q22	1072	1042	1016	1016
Total cold run time: 105070 ms
Total hot run time: 34788 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5135	5094	5127	5094
q2	250	325	232	232
q3	2207	2666	2318	2318
q4	1386	1774	1346	1346
q5	4227	4435	4664	4435
q6	217	178	142	142
q7	2071	1985	1870	1870
q8	2614	2714	2621	2621
q9	7491	7301	7294	7294
q10	3110	3380	3008	3008
q11	586	538	552	538
q12	722	767	626	626
q13	3474	3943	3355	3355
q14	289	309	336	309
q15	526	513	529	513
q16	1105	1108	1067	1067
q17	1185	1576	1390	1390
q18	7919	7746	7613	7613
q19	858	921	1116	921
q20	2031	2096	1890	1890
q21	4986	4451	4290	4290
q22	1075	1043	1023	1023
Total cold run time: 53464 ms
Total hot run time: 51895 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189473 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d42c4bd01a234fdf2fc35f30308f607cabcc45e7, data reload: false

query1	1091	420	396	396
query2	6577	1695	1691	1691
query3	6760	229	222	222
query4	26357	23489	23555	23489
query5	4354	647	492	492
query6	363	246	232	232
query7	4655	509	308	308
query8	309	249	255	249
query9	8662	2635	2636	2635
query10	496	330	300	300
query11	15314	15093	14851	14851
query12	169	114	113	113
query13	1678	554	434	434
query14	10262	9233	9249	9233
query15	208	201	174	174
query16	7231	674	500	500
query17	1266	769	642	642
query18	2024	461	353	353
query19	222	217	187	187
query20	136	126	122	122
query21	219	137	127	127
query22	4025	4140	4063	4063
query23	33937	32979	32980	32979
query24	8357	2450	2455	2450
query25	661	503	434	434
query26	1251	275	160	160
query27	2742	505	366	366
query28	4259	2249	2227	2227
query29	813	612	490	490
query30	294	223	194	194
query31	919	800	800	800
query32	79	78	74	74
query33	585	383	327	327
query34	798	845	527	527
query35	825	813	750	750
query36	980	1025	896	896
query37	116	104	80	80
query38	3538	3494	3554	3494
query39	1494	1447	1418	1418
query40	223	132	145	132
query41	73	64	65	64
query42	141	116	112	112
query43	521	502	467	467
query44	1355	866	858	858
query45	189	180	177	177
query46	863	1020	663	663
query47	1800	1821	1746	1746
query48	407	425	343	343
query49	776	522	398	398
query50	663	700	412	412
query51	4133	3908	3953	3908
query52	113	111	106	106
query53	247	269	202	202
query54	607	600	535	535
query55	95	86	86	86
query56	324	311	315	311
query57	1196	1199	1111	1111
query58	282	271	278	271
query59	2582	2688	2530	2530
query60	348	333	330	330
query61	172	164	161	161
query62	813	732	666	666
query63	232	214	196	196
query64	4397	1169	878	878
query65	4026	3975	3969	3969
query66	1157	448	351	351
query67	15441	15378	15195	15195
query68	9180	963	582	582
query69	530	322	294	294
query70	1411	1318	1234	1234
query71	594	351	321	321
query72	6066	5022	5192	5022
query73	783	646	370	370
query74	9044	9074	9012	9012
query75	4445	3348	2859	2859
query76	3818	1232	741	741
query77	822	425	320	320
query78	9631	9860	8933	8933
query79	4504	854	571	571
query80	686	585	506	506
query81	478	274	230	230
query82	468	166	131	131
query83	308	273	255	255
query84	306	121	97	97
query85	888	478	439	439
query86	353	305	313	305
query87	3862	3750	3628	3628
query88	3405	2246	2240	2240
query89	450	335	304	304
query90	1941	224	219	219
query91	192	170	136	136
query92	83	70	68	68
query93	2862	1005	654	654
query94	701	413	363	363
query95	402	322	327	322
query96	511	566	280	280
query97	2939	2973	2856	2856
query98	250	218	253	218
query99	1479	1432	1306	1306
Total cold run time: 280594 ms
Total hot run time: 189473 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d42c4bd01a234fdf2fc35f30308f607cabcc45e7, data reload: false

query1	0.08	0.01	0.01
query2	0.11	0.01	0.00
query3	0.26	0.00	0.01
query4	1.74	0.01	0.01
query5	0.29	0.01	0.00
query6	1.66	0.01	0.00
query7	0.04	0.01	0.01
query8	0.07	0.00	0.01
query9	0.62	0.00	0.01
query10	0.60	0.01	0.00
query11	0.17	0.00	0.00
query12	0.17	0.00	0.01
query13	0.63	0.01	0.01
query14	1.06	0.01	0.01
query15	0.91	0.00	0.01
query16	0.39	0.00	0.01
query17	1.07	0.00	0.00
query18	0.22	0.00	0.01
query19	2.00	0.00	0.00
query20	0.02	0.00	0.00
query21	15.39	0.01	0.00
query22	5.94	0.00	0.00
query23	15.78	0.00	0.00
query24	1.45	0.00	0.00
query25	0.22	0.01	0.00
query26	0.18	0.00	0.00
query27	0.10	0.01	0.01
query28	1.18	0.00	0.01
query29	13.28	0.00	0.00
query30	0.31	0.00	0.00
query31	2.30	0.00	0.01
query32	5.92	0.00	0.00
query33	4.38	0.00	0.01
query34	7.25	0.01	0.01
query35	6.06	0.01	0.00
query36	0.70	0.00	0.00
query37	0.11	0.00	0.00
query38	0.08	0.00	0.00
query39	0.05	0.00	0.01
query40	0.19	0.01	0.01
query41	0.09	0.01	0.01
query42	0.06	0.01	0.00
query43	0.05	0.00	0.01
Total cold run time: 93.18 s
Total hot run time: 0.09 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.21% (17429/33385)
Line Coverage 37.43% (158379/423156)
Region Coverage 31.98% (120714/377465)
Branch Coverage 33.37% (52992/158822)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.86% (23132/32646)
Line Coverage 57.23% (241578/422144)
Region Coverage 52.63% (201303/382458)
Branch Coverage 54.27% (86567/159523)

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@dqz123
Copy link

dqz123 commented Sep 16, 2025

run vault_p0

morningman
morningman previously approved these changes Sep 21, 2025
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
Copy link
Contributor

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 84.02% (1541/1834)
Line Coverage 68.01% (27619/40610)
Region Coverage 68.42% (13593/19868)
Branch Coverage 58.71% (7258/12362)

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.34% (17595/33618)
Line Coverage 37.56% (159700/425199)
Region Coverage 32.11% (121669/378959)
Branch Coverage 33.46% (53335/159399)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.14% (23375/32857)
Line Coverage 57.51% (244020/424306)
Region Coverage 52.95% (203383/384133)
Branch Coverage 54.52% (87314/160136)

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 21, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 376806f into apache:master Sep 29, 2025
30 of 33 checks passed
morningman pushed a commit to morningman/doris that referenced this pull request Oct 20, 2025
Make the `read slice size` variable configurable to reduce the impact
caused by read amplification.
Co-authored-by: garenshi <garenshi@tencnet.com>
github-actions bot pushed a commit that referenced this pull request Oct 20, 2025
### What problem does this PR solve?

Make the `read slice size` variable configurable to reduce the impact
caused by read amplification.
Co-authored-by: garenshi <garenshi@tencnet.com>
yiguolei pushed a commit that referenced this pull request Oct 22, 2025
#57160)

Cherry-picked from #55936

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencnet.com>
morrySnow pushed a commit that referenced this pull request Oct 30, 2025
…#57272 (#57159)

bp #55936 #57272

---------

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencnet.com>
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
…he#55936) (apache#5214)

close apache#4875

---------

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencnet.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants