Skip to content

[fix](merge-cloud) Ignore in_memory property in cloud mode#30702

Merged
dataroaring merged 1 commit into
apache:masterfrom
w41ter:fix/cloud_mode_ignore_in_memory
Feb 2, 2024
Merged

[fix](merge-cloud) Ignore in_memory property in cloud mode#30702
dataroaring merged 1 commit into
apache:masterfrom
w41ter:fix/cloud_mode_ignore_in_memory

Conversation

@w41ter
Copy link
Copy Markdown
Contributor

@w41ter w41ter commented Feb 1, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Feb 1, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 37343 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

------ Round 1 ----------------------------------
q1	17852	4586	4612	4586
q2	2861	146	138	138
q3	11414	947	944	944
q4	5249	770	759	759
q5	7852	2972	3019	2972
q6	183	124	124	124
q7	1144	742	754	742
q8	9709	2093	2052	2052
q9	7278	6350	6271	6271
q10	8110	2466	2425	2425
q11	420	206	214	206
q12	773	271	280	271
q13	18030	3318	3288	3288
q14	268	263	255	255
q15	544	502	486	486
q16	457	397	405	397
q17	952	507	598	507
q18	6816	5860	5858	5858
q19	1621	1435	1475	1435
q20	557	323	343	323
q21	6767	3017	3003	3003
q22	795	301	309	301
Total cold run time: 109652 ms
Total hot run time: 37343 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4507	4438	4378	4378
q2	334	232	245	232
q3	2999	2814	2831	2814
q4	1801	1616	1608	1608
q5	5227	5219	5300	5219
q6	189	115	117	115
q7	2149	1804	1754	1754
q8	3138	3262	3287	3262
q9	8383	8333	8332	8332
q10	5974	3615	3616	3615
q11	572	467	468	467
q12	730	581	578	578
q13	8380	3104	3092	3092
q14	277	264	267	264
q15	529	499	488	488
q16	545	483	480	480
q17	1868	1686	1732	1686
q18	7920	7760	7588	7588
q19	7567	1523	1519	1519
q20	2140	1914	1897	1897
q21	4657	4629	4615	4615
q22	581	473	506	473
Total cold run time: 70467 ms
Total hot run time: 54476 ms

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 1, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 174793 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

query1	940	336	329	329
query2	6543	2040	1926	1926
query3	6711	199	197	197
query4	31983	22048	21944	21944
query5	4465	351	418	351
query6	251	162	151	151
query7	4601	261	259	259
query8	251	170	174	170
query9	8952	2295	2282	2282
query10	405	191	217	191
query11	18620	15413	15398	15398
query12	117	71	67	67
query13	1614	373	376	373
query14	10163	7375	7143	7143
query15	229	176	184	176
query16	7359	270	256	256
query17	1842	485	465	465
query18	1933	272	250	250
query19	180	130	131	130
query20	74	68	73	68
query21	198	131	126	126
query22	4949	4678	4685	4678
query23	31401	30178	30351	30178
query24	11622	2692	2797	2692
query25	518	300	314	300
query26	1468	135	141	135
query27	2491	285	282	282
query28	7154	1869	1862	1862
query29	1457	611	592	592
query30	284	133	139	133
query31	931	701	690	690
query32	91	55	47	47
query33	714	217	201	201
query34	1096	457	462	457
query35	876	771	763	763
query36	1266	1236	1169	1169
query37	174	59	58	58
query38	3299	3183	3170	3170
query39	1288	1252	1242	1242
query40	328	85	83	83
query41	36	37	34	34
query42	85	76	83	76
query43	540	499	476	476
query44	1108	696	713	696
query45	190	178	178	178
query46	1040	633	625	625
query47	1516	1432	1508	1432
query48	397	302	308	302
query49	1234	280	284	280
query50	697	310	307	307
query51	5168	5150	5129	5129
query52	107	73	76	73
query53	312	258	264	258
query54	277	179	189	179
query55	77	74	75	74
query56	177	156	166	156
query57	1047	887	886	886
query58	191	148	160	148
query59	2409	2435	2238	2238
query60	195	177	174	174
query61	82	83	81	81
query62	655	374	347	347
query63	292	269	262	262
query64	6371	3629	3497	3497
query65	3249	3225	3228	3225
query66	1393	324	312	312
query67	14606	14316	14104	14104
query68	4345	499	490	490
query69	432	292	292	292
query70	1497	1538	1504	1504
query71	287	216	203	203
query72	5377	3142	2828	2828
query73	690	319	328	319
query74	6722	6215	6326	6215
query75	2985	2301	2247	2247
query76	2508	1001	891	891
query77	445	239	224	224
query78	9040	8938	8604	8604
query79	2437	494	480	480
query80	2047	313	304	304
query81	528	198	194	194
query82	763	78	77	77
query83	298	115	113	113
query84	275	70	75	70
query85	2388	333	323	323
query86	488	403	409	403
query87	3500	3326	3346	3326
query88	3901	2177	2162	2162
query89	429	343	350	343
query90	2071	182	177	177
query91	159	115	117	115
query92	57	43	41	41
query93	2505	424	418	418
query94	1526	153	154	153
query95	493	442	440	440
query96	622	318	319	318
query97	4210	4102	4149	4102
query98	195	192	191	191
query99	1119	698	693	693
Total cold run time: 286628 ms
Total hot run time: 174793 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 30.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.05
query4	1.67	0.10	0.10
query5	0.52	0.51	0.51
query6	1.18	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.50	0.49
query10	0.53	0.55	0.59
query11	0.12	0.08	0.09
query12	0.10	0.09	0.09
query13	0.62	0.61	0.61
query14	0.77	0.80	0.80
query15	0.79	0.78	0.78
query16	0.38	0.38	0.39
query17	0.97	1.00	1.05
query18	0.18	0.28	0.22
query19	1.79	1.78	1.75
query20	0.02	0.01	0.01
query21	15.41	0.56	0.56
query22	2.33	2.85	1.63
query23	17.23	0.88	0.75
query24	2.47	1.20	1.27
query25	0.30	0.36	0.12
query26	0.46	0.14	0.13
query27	0.05	0.04	0.04
query28	12.34	0.84	0.83
query29	12.52	3.22	3.12
query30	0.70	0.55	0.50
query31	2.79	0.34	0.34
query32	3.38	0.47	0.47
query33	3.22	3.24	3.27
query34	15.68	4.26	4.35
query35	4.30	4.25	4.28
query36	1.09	1.05	1.08
query37	0.06	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.13
query41	0.06	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.23 s
Total hot run time: 30.9 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@hello-stephen
Copy link
Copy Markdown
Contributor

run performance

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 36916 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

------ Round 1 ----------------------------------
q1	17691	5322	4466	4466
q2	2039	143	134	134
q3	10708	917	910	910
q4	4714	796	806	796
q5	7707	2837	2814	2814
q6	185	117	118	117
q7	1096	723	710	710
q8	9629	2035	2049	2035
q9	7250	6319	6289	6289
q10	8106	2505	2427	2427
q11	416	198	206	198
q12	769	269	281	269
q13	18016	3307	3248	3248
q14	272	241	255	241
q15	527	498	491	491
q16	490	416	395	395
q17	949	516	518	516
q18	6843	5867	5916	5867
q19	1575	1475	1375	1375
q20	580	334	308	308
q21	6664	3029	3097	3029
q22	791	310	281	281
Total cold run time: 107017 ms
Total hot run time: 36916 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4519	4382	4333	4333
q2	328	232	225	225
q3	3001	2845	2837	2837
q4	1855	1608	1631	1608
q5	5212	5227	5187	5187
q6	192	113	115	113
q7	2171	1747	1737	1737
q8	3138	3206	3281	3206
q9	8357	8271	8313	8271
q10	5807	3503	3569	3503
q11	531	449	441	441
q12	765	556	565	556
q13	11541	3066	3088	3066
q14	281	269	255	255
q15	555	487	491	487
q16	525	470	463	463
q17	1852	1700	1678	1678
q18	8023	7591	7691	7591
q19	7986	1509	1547	1509
q20	2097	1928	1898	1898
q21	4766	4559	4610	4559
q22	552	478	462	462
Total cold run time: 74054 ms
Total hot run time: 53985 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 175922 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

query1	928	342	331	331
query2	6573	2024	1872	1872
query3	6707	200	191	191
query4	32370	22186	22179	22179
query5	4474	361	416	361
query6	251	165	160	160
query7	4602	274	258	258
query8	254	178	181	178
query9	9041	2315	2305	2305
query10	409	209	202	202
query11	18456	15529	15442	15442
query12	120	67	69	67
query13	1617	363	378	363
query14	10225	7241	7197	7197
query15	222	187	187	187
query16	7302	259	249	249
query17	1836	502	486	486
query18	1917	261	249	249
query19	181	134	128	128
query20	75	69	70	69
query21	189	136	132	132
query22	4953	4788	4546	4546
query23	31133	30380	30331	30331
query24	8152	2770	2808	2770
query25	514	321	301	301
query26	723	141	143	141
query27	2220	290	299	290
query28	5266	1903	1877	1877
query29	935	643	620	620
query30	282	131	136	131
query31	950	713	721	713
query32	93	57	50	50
query33	494	204	206	204
query34	815	449	464	449
query35	871	774	744	744
query36	1294	1191	1262	1191
query37	96	59	59	59
query38	3317	3163	3173	3163
query39	1317	1260	1242	1242
query40	191	90	89	89
query41	38	34	32	32
query42	95	82	84	82
query43	522	474	470	470
query44	1064	702	729	702
query45	195	177	177	177
query46	1048	651	642	642
query47	1591	1454	1556	1454
query48	402	343	299	299
query49	1132	287	281	281
query50	673	308	303	303
query51	5253	5152	5132	5132
query52	87	83	76	76
query53	326	254	262	254
query54	222	185	193	185
query55	81	76	77	76
query56	175	179	158	158
query57	977	909	912	909
query58	184	159	154	154
query59	2330	2282	2299	2282
query60	200	178	176	176
query61	83	82	81	81
query62	607	362	344	344
query63	274	270	271	270
query64	4739	3646	3469	3469
query65	3239	3223	3212	3212
query66	940	322	315	315
query67	14566	14253	14411	14253
query68	4152	519	522	519
query69	431	313	306	306
query70	1548	1540	1561	1540
query71	286	224	212	212
query72	5547	3089	2831	2831
query73	701	314	324	314
query74	6724	6319	6332	6319
query75	2998	2295	2332	2295
query76	2557	982	915	915
query77	368	237	228	228
query78	9354	8822	8631	8631
query79	2332	524	498	498
query80	1505	326	324	324
query81	505	195	203	195
query82	636	82	85	82
query83	223	122	120	120
query84	227	76	72	72
query85	1765	351	329	329
query86	520	403	401	401
query87	3499	3257	3274	3257
query88	3656	2197	2171	2171
query89	432	367	349	349
query90	1967	189	188	188
query91	152	114	126	114
query92	54	44	45	44
query93	2539	469	430	430
query94	1203	167	161	161
query95	496	456	451	451
query96	646	330	337	330
query97	4220	4109	4150	4109
query98	202	188	186	186
query99	1124	696	711	696
Total cold run time: 274977 ms
Total hot run time: 175922 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 31.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.22	0.06	0.06
query4	1.67	0.11	0.11
query5	0.52	0.52	0.52
query6	1.22	0.67	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.56	0.48	0.50
query10	0.55	0.53	0.55
query11	0.13	0.09	0.09
query12	0.11	0.09	0.10
query13	0.60	0.62	0.58
query14	0.79	0.81	0.78
query15	0.79	0.78	0.76
query16	0.37	0.38	0.39
query17	1.01	1.04	1.03
query18	0.22	0.28	0.25
query19	1.83	1.80	1.76
query20	0.01	0.02	0.02
query21	15.41	0.56	0.55
query22	2.64	2.66	2.16
query23	17.31	0.75	0.79
query24	2.34	1.33	1.14
query25	0.30	0.22	0.12
query26	0.54	0.14	0.15
query27	0.06	0.04	0.06
query28	11.99	0.84	0.83
query29	12.52	3.17	3.17
query30	0.62	0.55	0.53
query31	2.77	0.34	0.35
query32	3.36	0.48	0.48
query33	3.20	3.25	3.26
query34	15.75	4.31	4.24
query35	4.30	4.30	4.30
query36	1.11	1.06	1.06
query37	0.07	0.05	0.06
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.13	0.12
query41	0.07	0.01	0.01
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.39 s
Total hot run time: 31.48 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 17cf09461c5b32d0b75ee2c9b25e6997914fa1bc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 2, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 2, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit bd66f12 into apache:master Feb 2, 2024
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants