Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](nereids) speedup sql cache with use variable as partition predicate #37915

Merged
merged 3 commits into from
Jul 16, 2024
Merged

[enhancement](nereids) speedup sql cache with use variable as partition predicate #37915

merged 3 commits into from
Jul 16, 2024

Conversation

924060929
Copy link
Contributor

@924060929 924060929 commented Jul 16, 2024 •
edited
Loading

Proposed changes

follow up #37090

support reuse sql cache when use variable as partition predicate and variable change:

set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40038 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12e1d342b0eb3ec11eb332926d1186d9a8ad6b37, data reload: false

------ Round 1 ----------------------------------
q1	17615	4443	4271	4271
q2	2023	192	189	189
q3	10445	1241	1147	1147
q4	10192	761	760	760
q5	7565	2715	2608	2608
q6	221	137	137	137
q7	948	610	611	610
q8	9212	2079	2089	2079
q9	8922	6567	6562	6562
q10	8876	3813	3793	3793
q11	473	241	246	241
q12	419	232	234	232
q13	17770	2989	2994	2989
q14	287	246	247	246
q15	525	480	494	480
q16	493	391	378	378
q17	975	670	715	670
q18	8075	7538	7487	7487
q19	4810	1405	1403	1403
q20	694	328	322	322
q21	5103	3147	3387	3147
q22	352	292	287	287
Total cold run time: 115995 ms
Total hot run time: 40038 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4283	4299	4283
q2	368	276	268	268
q3	3014	2791	2933	2791
q4	2023	1735	1712	1712
q5	5720	5530	5539	5530
q6	219	135	130	130
q7	2184	1862	1891	1862
q8	3272	3421	3439	3421
q9	8853	8822	8922	8822
q10	4092	3973	3768	3768
q11	577	501	500	500
q12	824	654	618	618
q13	16164	3184	3181	3181
q14	330	290	308	290
q15	540	487	477	477
q16	487	447	441	441
q17	1824	1542	1508	1508
q18	8054	7832	7743	7743
q19	2617	1617	1705	1617
q20	2139	1866	1868	1866
q21	5069	5100	4854	4854
q22	608	522	512	512
Total cold run time: 73360 ms
Total hot run time: 56194 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173758 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12e1d342b0eb3ec11eb332926d1186d9a8ad6b37, data reload: false

query1	916	397	363	363
query2	6436	1916	1860	1860
query3	6639	207	222	207
query4	28495	17602	17218	17218
query5	3582	475	491	475
query6	290	168	170	168
query7	4603	297	292	292
query8	255	200	198	198
query9	8588	2396	2364	2364
query10	436	331	271	271
query11	12241	9945	10120	9945
query12	121	82	84	82
query13	1638	372	389	372
query14	10283	7194	7726	7194
query15	216	165	160	160
query16	7726	319	311	311
query17	1371	550	524	524
query18	1859	296	273	273
query19	195	150	159	150
query20	93	84	85	84
query21	210	134	134	134
query22	4342	4141	4082	4082
query23	34131	33878	33658	33658
query24	11319	2961	2949	2949
query25	630	434	399	399
query26	1209	153	154	153
query27	2456	281	294	281
query28	7404	2032	2019	2019
query29	885	667	646	646
query30	255	157	151	151
query31	996	777	751	751
query32	100	59	59	59
query33	783	300	292	292
query34	1004	501	521	501
query35	724	619	649	619
query36	1157	1016	986	986
query37	144	81	86	81
query38	2964	2837	2825	2825
query39	923	820	825	820
query40	210	122	120	120
query41	49	45	43	43
query42	119	101	107	101
query43	527	458	459	458
query44	1249	737	738	737
query45	202	159	160	159
query46	1090	736	733	733
query47	1880	1771	1782	1771
query48	372	303	299	299
query49	833	416	424	416
query50	786	404	407	404
query51	6926	6880	6775	6775
query52	102	98	93	93
query53	359	303	301	301
query54	878	451	452	451
query55	75	77	77	77
query56	321	265	270	265
query57	1130	1038	1057	1038
query58	244	254	256	254
query59	2832	2730	2600	2600
query60	306	279	293	279
query61	96	93	92	92
query62	794	636	656	636
query63	324	295	308	295
query64	9389	2289	1746	1746
query65	3388	3139	3146	3139
query66	763	330	338	330
query67	15431	15155	15097	15097
query68	5822	540	542	540
query69	675	450	374	374
query70	1209	1178	1119	1119
query71	454	289	285	285
query72	7542	5640	5512	5512
query73	776	331	329	329
query74	6183	5876	5692	5692
query75	3513	2691	2719	2691
query76	3220	1022	905	905
query77	615	296	310	296
query78	9642	8970	8945	8945
query79	1602	538	543	538
query80	1198	494	466	466
query81	590	220	225	220
query82	740	132	127	127
query83	268	172	165	165
query84	244	84	87	84
query85	1334	353	394	353
query86	462	327	331	327
query87	3271	3131	3139	3131
query88	4000	2471	2462	2462
query89	473	401	392	392
query90	1690	201	202	201
query91	131	101	110	101
query92	58	50	50	50
query93	2423	512	516	512
query94	1046	226	219	219
query95	415	317	327	317
query96	607	289	277	277
query97	3218	3018	3058	3018
query98	230	208	201	201
query99	1517	1287	1283	1283
Total cold run time: 283462 ms
Total hot run time: 173758 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12e1d342b0eb3ec11eb332926d1186d9a8ad6b37, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.48	0.49	0.49
query6	1.13	0.73	0.73
query7	0.01	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.50
query10	0.56	0.56	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.78	0.79
query15	0.86	0.82	0.82
query16	0.37	0.37	0.37
query17	1.01	1.03	1.03
query18	0.22	0.21	0.21
query19	1.81	1.74	1.85
query20	0.01	0.01	0.01
query21	15.41	0.75	0.65
query22	3.56	6.70	2.54
query23	18.28	1.38	1.24
query24	2.05	0.27	0.22
query25	0.16	0.09	0.09
query26	0.30	0.20	0.20
query27	0.45	0.22	0.22
query28	13.25	1.02	0.99
query29	12.59	3.36	3.30
query30	0.25	0.06	0.05
query31	2.87	0.38	0.38
query32	3.30	0.47	0.47
query33	2.89	2.87	2.91
query34	17.14	4.43	4.40
query35	4.50	4.43	4.45
query36	0.65	0.47	0.48
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 109.11 s
Total hot run time: 31.34 s

@924060929 924060929 mentioned this pull request Jul 16, 2024
Merged
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit 487e4ec into apache:master Jul 16, 2024
26 of 29 checks passed
@924060929 924060929 deleted the optimize_sql_cache_with_variable branch July 16, 2024 12:49
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@924060929 @seawinde
[enhancement](nereids) speedup sql cache with use variable as partiti…
d3c5d9c 
…on predicate (apache#37915)

support reuse sql cache when use variable as partition predicate and variable change:
```sql
set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;
```
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@924060929 @dataroaring
[enhancement](nereids) speedup sql cache with use variable as partiti…
0341571 
…on predicate (#37915)

support reuse sql cache when use variable as partition predicate and variable change:
```sql
set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;
```
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feiniaofeiafei feiniaofeiafei feiniaofeiafei approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@924060929 @doris-robot @feiniaofeiafei @morrySnow @dataroaring