Skip to content

[opt](load) disable enable_strict_consistency_dml by default in cloud mode#61814

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:opt_cloud_load_memory
Mar 30, 2026
Merged

[opt](load) disable enable_strict_consistency_dml by default in cloud mode#61814
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:opt_cloud_load_memory

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented Mar 27, 2026

Problem

enable_strict_consistency_dml is designed to handle multi-replica consistency for DML
operations. In cloud mode (store-compute separation), data has only a single copy, so
there is no multi-replica consistency concern. Keeping this variable true in cloud mode
causes unnecessary data shuffling overhead.

Solution

Override isEnableStrictConsistencyDml() to return false when running in cloud mode
(Config.isCloudMode()), regardless of the session variable's value. This follows the
same pattern as isDisableFileCache().

Update all call sites in RequestPropertyDeriver to use the getter instead of direct
field access so the cloud-mode check takes effect.

Tests

Added two unit tests in SessionVariablesTest:

  • Verify isEnableStrictConsistencyDml() returns false in cloud mode (even when field is true)
  • Verify isEnableStrictConsistencyDml() follows the field value in non-cloud mode

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Mar 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui sollhui changed the title [fix](cloud) disable enable_strict_consistency_dml by default in cloud mode [opt](load) disable enable_strict_consistency_dml by default in cloud mode Mar 27, 2026
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Mar 27, 2026

run buildall

liaoxin01
liaoxin01 previously approved these changes Mar 27, 2026
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2026
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@sollhui sollhui force-pushed the opt_cloud_load_memory branch from 88fccba to 398950e Compare March 27, 2026 10:07
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 27, 2026
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Mar 27, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 26284 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 398950ed3b23381e7403cc32fefbb75f1057d4ae, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17661	4459	4289	4289
q2	q3	10639	777	522	522
q4	4683	367	251	251
q5	7566	1203	1024	1024
q6	179	173	147	147
q7	775	834	659	659
q8	9337	1467	1338	1338
q9	4927	4749	4679	4679
q10	6326	1911	1618	1618
q11	464	265	246	246
q12	752	581	465	465
q13	18021	2670	1924	1924
q14	225	230	208	208
q15	q16	732	741	648	648
q17	733	864	419	419
q18	5896	5526	5196	5196
q19	1113	969	628	628
q20	537	479	375	375
q21	4468	1848	1396	1396
q22	351	303	252	252
Total cold run time: 95385 ms
Total hot run time: 26284 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4905	4658	4597	4597
q2	q3	3866	4367	3818	3818
q4	867	1236	805	805
q5	4139	4450	4448	4448
q6	188	182	139	139
q7	1783	1660	1545	1545
q8	2482	2648	2558	2558
q9	7585	7458	7324	7324
q10	3759	4047	3643	3643
q11	506	431	419	419
q12	511	618	469	469
q13	2455	3035	2174	2174
q14	325	400	285	285
q15	q16	724	787	736	736
q17	1180	1371	1511	1371
q18	7306	6935	6744	6744
q19	889	892	898	892
q20	2051	2127	2006	2006
q21	3996	3464	3332	3332
q22	612	479	397	397
Total cold run time: 50129 ms
Total hot run time: 47702 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 168803 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 398950ed3b23381e7403cc32fefbb75f1057d4ae, data reload: false

query5	4318	634	503	503
query6	337	227	202	202
query7	4208	478	264	264
query8	340	248	219	219
query9	8649	2734	2731	2731
query10	499	387	345	345
query11	6994	5112	4846	4846
query12	181	127	120	120
query13	1305	491	357	357
query14	5877	3729	3478	3478
query14_1	2850	2843	2864	2843
query15	203	203	176	176
query16	974	466	459	459
query17	899	744	621	621
query18	2453	465	356	356
query19	218	216	191	191
query20	145	127	129	127
query21	215	134	108	108
query22	13270	14050	14578	14050
query23	16790	16475	15940	15940
query23_1	15994	15653	15760	15653
query24	7136	1615	1216	1216
query24_1	1231	1251	1263	1251
query25	601	471	409	409
query26	1239	263	152	152
query27	2753	484	294	294
query28	4469	1863	1863	1863
query29	852	573	490	490
query30	296	228	189	189
query31	1032	939	870	870
query32	84	68	71	68
query33	516	356	281	281
query34	892	895	523	523
query35	643	696	592	592
query36	1066	1162	1005	1005
query37	137	94	82	82
query38	2963	2925	2941	2925
query39	860	827	821	821
query39_1	804	782	801	782
query40	236	150	133	133
query41	62	58	57	57
query42	260	257	257	257
query43	246	251	219	219
query44	
query45	191	188	180	180
query46	913	1004	623	623
query47	2115	2153	2459	2153
query48	309	327	230	230
query49	637	466	378	378
query50	717	278	209	209
query51	4066	4054	3981	3981
query52	253	263	254	254
query53	290	337	288	288
query54	300	277	272	272
query55	95	87	85	85
query56	323	319	322	319
query57	1935	1895	1720	1720
query58	280	270	271	270
query59	2781	2966	2729	2729
query60	344	327	316	316
query61	158	157	152	152
query62	626	595	545	545
query63	321	306	287	287
query64	5078	1290	1029	1029
query65	
query66	1476	453	352	352
query67	24132	24384	24064	24064
query68	
query69	424	325	280	280
query70	983	977	954	954
query71	338	315	294	294
query72	2807	2688	2440	2440
query73	543	563	317	317
query74	9594	9522	9407	9407
query75	2872	2759	2456	2456
query76	2268	1065	687	687
query77	368	374	326	326
query78	10887	11058	10487	10487
query79	2747	782	595	595
query80	1774	631	558	558
query81	541	260	223	223
query82	992	151	115	115
query83	332	285	251	251
query84	295	121	103	103
query85	909	518	458	458
query86	421	298	259	259
query87	3151	3128	3012	3012
query88	3597	2653	2657	2653
query89	424	374	350	350
query90	2026	176	177	176
query91	165	170	143	143
query92	73	74	74	74
query93	1173	864	504	504
query94	660	316	285	285
query95	590	410	321	321
query96	653	519	233	233
query97	2449	2483	2388	2388
query98	247	223	219	219
query99	997	1015	914	914
Total cold run time: 252260 ms
Total hot run time: 168803 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉
Increment coverage report
Complete coverage report

@sollhui sollhui force-pushed the opt_cloud_load_memory branch from 398950e to f94dc5c Compare March 30, 2026 03:37
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Mar 30, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 26669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f94dc5cd99fd0be9fcbc3f272f1d5e8a14735e3c, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17650	4502	4288	4288
q2	q3	10644	790	526	526
q4	4676	364	248	248
q5	7677	1210	1013	1013
q6	181	178	150	150
q7	781	867	699	699
q8	9303	1492	1389	1389
q9	4885	4733	4722	4722
q10	6313	1901	1664	1664
q11	479	272	250	250
q12	748	588	475	475
q13	18055	2730	1979	1979
q14	236	225	209	209
q15	q16	744	731	665	665
q17	744	841	447	447
q18	6196	5323	5134	5134
q19	1112	1024	661	661
q20	558	486	373	373
q21	4523	1871	1448	1448
q22	447	489	329	329
Total cold run time: 95952 ms
Total hot run time: 26669 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4704	4556	4542	4542
q2	q3	3895	4453	3925	3925
q4	905	1220	833	833
q5	4087	4474	4374	4374
q6	200	182	148	148
q7	1785	1671	1524	1524
q8	2525	2742	2580	2580
q9	7694	7338	7584	7338
q10	3768	4032	3601	3601
q11	531	472	441	441
q12	509	630	473	473
q13	2557	3210	2369	2369
q14	293	311	288	288
q15	q16	733	788	748	748
q17	1171	1410	1392	1392
q18	7356	6876	6694	6694
q19	947	945	965	945
q20	2045	2141	1988	1988
q21	4101	3530	3427	3427
q22	466	425	387	387
Total cold run time: 50272 ms
Total hot run time: 48017 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 169665 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f94dc5cd99fd0be9fcbc3f272f1d5e8a14735e3c, data reload: false

query5	4352	649	493	493
query6	326	229	214	214
query7	4195	470	266	266
query8	329	245	222	222
query9	8710	2721	2724	2721
query10	516	391	357	357
query11	6887	5112	4893	4893
query12	178	131	128	128
query13	1295	481	349	349
query14	5701	3787	3608	3608
query14_1	2920	2874	2864	2864
query15	209	193	178	178
query16	1012	474	490	474
query17	864	724	607	607
query18	2431	440	345	345
query19	219	209	179	179
query20	132	127	127	127
query21	214	133	110	110
query22	13196	14203	14917	14203
query23	16788	16180	16028	16028
query23_1	16277	15909	15660	15660
query24	7208	1675	1231	1231
query24_1	1211	1232	1243	1232
query25	532	458	416	416
query26	1233	268	152	152
query27	2772	484	297	297
query28	4450	1848	1870	1848
query29	856	582	472	472
query30	295	217	205	205
query31	1014	958	891	891
query32	81	73	70	70
query33	516	338	296	296
query34	894	886	534	534
query35	632	668	591	591
query36	1090	1103	950	950
query37	134	108	87	87
query38	2936	2896	2833	2833
query39	861	842	841	841
query39_1	797	801	794	794
query40	234	160	140	140
query41	77	68	59	59
query42	258	260	256	256
query43	242	267	232	232
query44	
query45	193	184	195	184
query46	890	997	615	615
query47	2134	2129	2008	2008
query48	308	317	233	233
query49	637	479	411	411
query50	726	291	222	222
query51	4026	4083	3962	3962
query52	266	275	259	259
query53	293	342	286	286
query54	303	287	268	268
query55	95	93	86	86
query56	334	321	322	321
query57	1953	1631	1722	1631
query58	285	284	275	275
query59	2826	3007	2753	2753
query60	354	338	329	329
query61	160	156	163	156
query62	645	590	530	530
query63	319	292	279	279
query64	5071	1320	1030	1030
query65	
query66	1498	482	385	385
query67	24279	24380	24180	24180
query68	
query69	432	339	306	306
query70	991	971	963	963
query71	352	322	329	322
query72	3016	2889	2649	2649
query73	554	562	340	340
query74	9615	9610	9401	9401
query75	2909	2788	2511	2511
query76	2283	1054	694	694
query77	369	385	316	316
query78	11032	11160	10459	10459
query79	1163	780	591	591
query80	1320	636	539	539
query81	546	260	226	226
query82	991	156	122	122
query83	338	280	257	257
query84	305	113	102	102
query85	948	506	470	470
query86	417	303	299	299
query87	3146	3196	3026	3026
query88	3613	2664	2648	2648
query89	427	381	355	355
query90	2029	187	187	187
query91	173	168	141	141
query92	80	78	73	73
query93	1032	892	502	502
query94	679	346	288	288
query95	589	423	317	317
query96	649	525	225	225
query97	2454	2468	2410	2410
query98	243	230	231	230
query99	1025	977	917	917
Total cold run time: 250525 ms
Total hot run time: 169665 ms

Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2026
@liaoxin01 liaoxin01 merged commit 9443e31 into apache:master Mar 30, 2026
28 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Mar 31, 2026
…ult in cloud mode (#61892)

pick #61814

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Mar 31, 2026
… mode (apache#61814)

## Problem
`enable_strict_consistency_dml` is designed to handle multi-replica
consistency for DML
operations. In cloud mode (store-compute separation), data has only a
single copy, so
there is no multi-replica consistency concern. Keeping this variable
`true` in cloud mode
causes unnecessary data shuffling overhead.

## Solution
Override `isEnableStrictConsistencyDml()` to return `false` when running
in cloud mode
(`Config.isCloudMode()`), regardless of the session variable's value.
This follows the
same pattern as `isDisableFileCache()`.

Update all call sites in `RequestPropertyDeriver` to use the getter
instead of direct
field access so the cloud-mode check takes effect.

## Tests
Added two unit tests in `SessionVariablesTest`:
- Verify `isEnableStrictConsistencyDml()` returns `false` in cloud mode
(even when field is `true`)
- Verify `isEnableStrictConsistencyDml()` follows the field value in
non-cloud mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.x dev/4.0.x-conflict dev/4.1.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants