Skip to content

[opt](hive)Add a variable to control whether to use the cache. #49201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

wuwenchi
Copy link
Contributor

What problem does this PR solve?

Problem Summary:

Add a variable to control whether to use the cache, which makes it convenient to eliminate the influence of the cache during testing.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

run buildall

1 similar comment
@wuwenchi
Copy link
Contributor Author

run buildall

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32115 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 751099d818c16cdc4f57b78245f640a97e0f9541, data reload: false

------ Round 1 ----------------------------------
q1	23833	5099	5004	5004
q2	2032	279	166	166
q3	10620	1217	712	712
q4	10214	1007	519	519
q5	7512	2305	2321	2305
q6	207	167	131	131
q7	919	759	612	612
q8	9308	1257	1033	1033
q9	4916	4822	4913	4822
q10	6797	2324	1885	1885
q11	470	270	259	259
q12	350	356	211	211
q13	17746	3632	3037	3037
q14	229	227	203	203
q15	518	483	490	483
q16	635	608	580	580
q17	557	839	361	361
q18	6954	6337	6216	6216
q19	1226	946	539	539
q20	322	328	199	199
q21	2856	2232	1858	1858
q22	1015	987	980	980
Total cold run time: 109236 ms
Total hot run time: 32115 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5138	5151	5121	5121
q2	226	321	234	234
q3	2140	2649	2290	2290
q4	1431	1874	1368	1368
q5	4317	4161	4352	4161
q6	216	169	128	128
q7	2029	1922	1788	1788
q8	2654	2523	2644	2523
q9	7261	7258	7251	7251
q10	3025	3238	2762	2762
q11	559	495	488	488
q12	705	777	629	629
q13	3567	3908	3283	3283
q14	275	304	264	264
q15	513	475	484	475
q16	639	676	671	671
q17	1144	1510	1440	1440
q18	7848	7700	7610	7610
q19	803	842	931	842
q20	2108	2131	1941	1941
q21	5801	5051	5152	5051
q22	1168	1127	1185	1127
Total cold run time: 53567 ms
Total hot run time: 51447 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192349 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 751099d818c16cdc4f57b78245f640a97e0f9541, data reload: false

query1	2681	1084	1041	1041
query2	6582	1939	1951	1939
query3	11037	4538	4649	4538
query4	25690	23632	23237	23237
query5	4699	645	472	472
query6	299	198	191	191
query7	4000	515	295	295
query8	304	255	235	235
query9	8493	2634	2624	2624
query10	466	317	265	265
query11	15331	15229	14860	14860
query12	166	111	104	104
query13	1563	522	404	404
query14	9513	6394	6283	6283
query15	225	187	174	174
query16	7635	659	451	451
query17	1296	772	578	578
query18	1988	435	320	320
query19	192	180	200	180
query20	125	129	121	121
query21	210	136	112	112
query22	4422	4465	4436	4436
query23	34084	33335	33451	33335
query24	7952	2427	2439	2427
query25	512	459	419	419
query26	723	270	156	156
query27	2434	496	333	333
query28	4342	2446	2432	2432
query29	644	575	479	479
query30	278	220	187	187
query31	889	887	789	789
query32	73	65	61	61
query33	560	358	314	314
query34	794	874	521	521
query35	821	836	787	787
query36	1007	1018	905	905
query37	116	103	88	88
query38	4212	4336	4105	4105
query39	1504	1448	1423	1423
query40	208	119	106	106
query41	54	51	51	51
query42	117	104	112	104
query43	504	541	487	487
query44	1339	810	835	810
query45	187	188	173	173
query46	854	1038	646	646
query47	1818	1904	1812	1812
query48	385	411	314	314
query49	701	506	406	406
query50	746	755	425	425
query51	4342	4305	4258	4258
query52	108	109	104	104
query53	231	259	197	197
query54	481	510	418	418
query55	85	82	90	82
query56	261	299	249	249
query57	1165	1199	1104	1104
query58	253	241	235	235
query59	2833	3007	2641	2641
query60	299	281	267	267
query61	138	126	128	126
query62	766	757	672	672
query63	231	189	196	189
query64	2935	1047	705	705
query65	4551	4429	4428	4428
query66	791	420	308	308
query67	16439	15658	15691	15658
query68	8975	867	497	497
query69	488	293	261	261
query70	1235	1127	1096	1096
query71	463	302	257	257
query72	5211	3578	3755	3578
query73	766	725	357	357
query74	9009	9148	8876	8876
query75	4034	3178	2697	2697
query76	3628	1187	764	764
query77	804	365	285	285
query78	9817	10137	9274	9274
query79	1327	841	588	588
query80	616	520	459	459
query81	459	261	224	224
query82	192	126	91	91
query83	193	169	156	156
query84	389	98	74	74
query85	745	349	375	349
query86	325	299	285	285
query87	4498	4396	4546	4396
query88	2914	2291	2286	2286
query89	393	329	294	294
query90	2032	217	218	217
query91	144	134	108	108
query92	78	61	56	56
query93	1149	1047	581	581
query94	660	391	291	291
query95	358	280	265	265
query96	486	565	278	278
query97	3231	3394	3336	3336
query98	218	205	201	201
query99	1356	1394	1276	1276
Total cold run time: 276877 ms
Total hot run time: 192349 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 751099d818c16cdc4f57b78245f640a97e0f9541, data reload: false

query1	0.04	0.04	0.02
query2	0.12	0.10	0.10
query3	0.25	0.18	0.19
query4	1.59	0.19	0.19
query5	0.58	0.58	0.58
query6	1.20	0.71	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.58	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.61	0.59	0.60
query14	2.81	2.81	2.82
query15	0.95	0.86	0.84
query16	0.39	0.37	0.38
query17	1.05	1.08	1.03
query18	0.21	0.19	0.20
query19	1.91	2.07	1.87
query20	0.01	0.01	0.02
query21	15.36	0.88	0.54
query22	0.74	1.19	0.64
query23	14.99	1.39	0.64
query24	6.80	0.87	1.43
query25	0.49	0.32	0.10
query26	0.50	0.15	0.13
query27	0.06	0.05	0.05
query28	10.04	0.90	0.44
query29	12.56	4.04	3.37
query30	0.24	0.10	0.06
query31	2.82	0.57	0.39
query32	3.23	0.55	0.46
query33	2.99	3.07	2.99
query34	15.73	5.09	4.49
query35	4.54	4.51	4.57
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.48 s
Total hot run time: 31.31 s

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32481 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2796c62c28b633896aa15ac2e1780b31bc3414e, data reload: false

------ Round 1 ----------------------------------
q1	23942	5142	5082	5082
q2	2045	292	179	179
q3	10597	1253	719	719
q4	10280	1034	533	533
q5	8990	2399	2398	2398
q6	197	169	141	141
q7	936	764	625	625
q8	9334	1342	1123	1123
q9	5048	4877	4603	4603
q10	6833	2316	1915	1915
q11	482	268	261	261
q12	349	369	227	227
q13	17799	3694	3074	3074
q14	245	247	212	212
q15	532	471	496	471
q16	626	597	598	597
q17	593	876	340	340
q18	6937	6521	6262	6262
q19	1430	946	571	571
q20	321	332	197	197
q21	2948	2157	1975	1975
q22	1032	1010	976	976
Total cold run time: 111496 ms
Total hot run time: 32481 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5127	5167	5184	5167
q2	238	329	235	235
q3	2208	2634	2268	2268
q4	1431	1850	1367	1367
q5	4295	4294	4477	4294
q6	224	171	135	135
q7	2048	1912	1764	1764
q8	2601	2537	2513	2513
q9	7302	7067	7236	7067
q10	2975	3183	2767	2767
q11	581	522	502	502
q12	685	759	592	592
q13	3597	3929	3312	3312
q14	291	293	277	277
q15	521	478	496	478
q16	678	686	653	653
q17	1146	1571	1393	1393
q18	7705	7607	7327	7327
q19	840	930	1104	930
q20	2001	1987	1880	1880
q21	5556	4946	4651	4651
q22	1058	1036	1002	1002
Total cold run time: 53108 ms
Total hot run time: 50574 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2796c62c28b633896aa15ac2e1780b31bc3414e, data reload: false

query1	1023	487	464	464
query2	6539	1988	1929	1929
query3	6792	216	221	216
query4	26513	23428	23390	23390
query5	4346	652	505	505
query6	301	207	194	194
query7	4611	503	294	294
query8	297	278	243	243
query9	8629	2641	2626	2626
query10	478	307	255	255
query11	15502	15146	14998	14998
query12	150	106	103	103
query13	1657	522	387	387
query14	8764	6257	6309	6257
query15	205	183	163	163
query16	7134	617	468	468
query17	1189	681	536	536
query18	1938	411	320	320
query19	201	188	160	160
query20	127	124	116	116
query21	210	132	105	105
query22	4238	4176	4118	4118
query23	33981	33136	32979	32979
query24	8466	2387	2368	2368
query25	570	485	418	418
query26	1236	277	162	162
query27	2737	508	339	339
query28	4347	2421	2385	2385
query29	807	584	452	452
query30	289	219	191	191
query31	945	863	792	792
query32	77	67	75	67
query33	592	366	315	315
query34	792	863	505	505
query35	853	827	736	736
query36	961	966	911	911
query37	125	100	74	74
query38	4357	4297	4151	4151
query39	1437	1390	1413	1390
query40	214	117	104	104
query41	56	52	51	51
query42	121	106	104	104
query43	498	509	492	492
query44	1288	821	789	789
query45	177	172	167	167
query46	841	1016	614	614
query47	1721	1779	1691	1691
query48	395	439	308	308
query49	778	497	420	420
query50	682	727	405	405
query51	4154	4229	4151	4151
query52	108	106	93	93
query53	233	270	190	190
query54	483	502	412	412
query55	80	80	81	80
query56	280	268	254	254
query57	1136	1179	1071	1071
query58	244	235	236	235
query59	2694	2794	2740	2740
query60	285	274	263	263
query61	120	120	117	117
query62	812	728	699	699
query63	237	192	192	192
query64	4365	998	677	677
query65	4405	4367	4381	4367
query66	1147	456	301	301
query67	15699	15546	15363	15363
query68	8274	870	494	494
query69	503	307	264	264
query70	1197	1126	1159	1126
query71	456	297	289	289
query72	5794	3625	3760	3625
query73	809	742	345	345
query74	9026	8941	8722	8722
query75	3675	3132	2696	2696
query76	3621	1187	751	751
query77	779	366	287	287
query78	10057	10044	9344	9344
query79	2680	815	584	584
query80	694	522	444	444
query81	485	255	220	220
query82	459	128	101	101
query83	208	176	154	154
query84	295	97	77	77
query85	783	348	307	307
query86	381	302	279	279
query87	4397	4477	4334	4334
query88	3564	2253	2265	2253
query89	388	330	285	285
query90	1841	218	289	218
query91	146	144	114	114
query92	76	64	55	55
query93	1953	1081	567	567
query94	661	416	305	305
query95	356	269	254	254
query96	481	570	280	280
query97	3401	3340	3282	3282
query98	234	215	206	206
query99	1447	1394	1257	1257
Total cold run time: 275790 ms
Total hot run time: 185951 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2796c62c28b633896aa15ac2e1780b31bc3414e, data reload: false

query1	0.03	0.04	0.02
query2	0.13	0.11	0.10
query3	0.25	0.19	0.20
query4	1.59	0.19	0.19
query5	0.57	0.56	0.55
query6	1.21	0.70	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.53	0.52
query10	0.62	0.59	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.81	2.81	2.81
query15	0.92	0.87	0.85
query16	0.39	0.38	0.38
query17	1.06	1.02	1.02
query18	0.21	0.19	0.20
query19	2.10	1.92	1.86
query20	0.01	0.01	0.01
query21	15.39	0.90	0.54
query22	0.75	1.10	0.64
query23	15.11	1.42	0.66
query24	7.00	1.89	0.99
query25	0.50	0.22	0.20
query26	0.53	0.16	0.12
query27	0.05	0.05	0.05
query28	9.30	0.87	0.45
query29	12.54	4.02	3.31
query30	0.26	0.08	0.06
query31	2.83	0.60	0.39
query32	3.24	0.54	0.46
query33	2.97	3.04	3.02
query34	15.74	5.10	4.53
query35	4.58	4.56	4.54
query36	0.66	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.4 s
Total hot run time: 31.61 s

@wuwenchi
Copy link
Contributor Author

run p0

@wuwenchi
Copy link
Contributor Author

run cloud_p0

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 24, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants