Skip to content

[Pick](query-cache) some query-cache bugfix (#47883)(#47961)(#51202) #51204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 27, 2025
Merged

[Pick](query-cache) some query-cache bugfix (#47883)(#47961)(#51202) #51204

merged 6 commits into from
May 27, 2025

Conversation

924060929
Copy link
Contributor

cherry pick from #47883, #47961, #51202

HappenLee and others added 3 commits May 23, 2025 19:40
@HappenLee @924060929
[Bug](cache) Fix query cache report error message (apache#47883)
3f5c5bf 
the plan：
```
 QUERY_CACHE:                                                                                                                                                                                                                                                                                                                           |
|     CACHE_NODE_ID: 2                                                                                                                                                                                                                                                                                                                     |
|     DIGEST: 557413bec3209f50fd640eb3b0534c12ccd9df35f9d4c86620416936985e2679                                                                                                                                                                                                                                                             |
|                                                                                                                                                                                                                                                                                                                                          |
|   STREAM DATA SINK                                                                                                                                                                                                                                                                                                                       |
|     EXCHANGE ID: 03                                                                                                                                                                                                                                                                                                                      |
|     UNPARTITIONED                                                                                                                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                                                                                                                          |
|   2:VAGGREGATE (merge finalize)(273)                                                                                                                                                                                                                                                                                                     |
|   |  output: min(partial_min(pk)[apache#6])[apache#9], max(partial_max(pk)[apache#7])[apache#10]                                                                                                                                                                                                                                                                 |
|   |  group by: pk[apache#5]                                                                                                                                                                                                                                                                                                                    |
|   |  sortByGroupKey:false                                                                                                                                                                                                                                                                                                                |
|   |  cardinality=0                                                                                                                                                                                                                                                                                                                       |
|   |  final projections: field1[apache#9], field2[apache#10], pk[apache#8]                                                                                                                                                                                                                                                                                  |
|   |  final project output tuple id: 4                                                                                                                                                                                                                                                                                                    |
|   |  distribute expr lists: pk[apache#5]                                                                                                                                                                                                                                                                                                       |
|   |                                                                                                                                                                                                                                                                                                                                      |
|   1:VAGGREGATE (update serialize)(268)                                                                                                                                                                                                                                                                                                   |
|   |  output: partial_min(pk[apache#4])[apache#6], partial_max(pk[apache#4])[apache#7]                                                                                                                                                                                                                                                                            |
|   |  group by: pk[apache#4]                                                                                                                                                                                                                                                                                                                    |
|   |  sortByGroupKey:false                                                                                                                                                                                                                                                                                                                |
|   |  cardinality=0                                                                                                                                                                                                                                                                                                                       |
|   |  distribute expr lists: pk[apache#4]                                                                                                                                                                                                                                                                                                       |
|   |
```

FE choose the id agg 2, but BE still use id agg 1. BE should support the case

(cherry picked from commit c58fc18)
@HappenLee @924060929
[Fix](QueryCache) Fix cache order different may cause error (apache#4…
92b0663 
…7961)

the origin judge may cause the need reorder judge error:
```
   bool need_reorder = _slot_orders.size() != hit_cache_slot_orders->size();
        if (!need_reorder) {
            for (int i = 0; i < _slot_orders.size(); ++i) {
                need_reorder = _slot_orders[i] != (*hit_cache_slot_orders)[i];
            }
        }
```

(cherry picked from commit e9a34e4)
@924060929
fix
36d8bcc
@924060929 924060929 requested a review from dataroaring as a code owner May 23, 2025 11:43
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10861/26443)
Line Coverage 31.87% (92688/290862)
Region Coverage 30.96% (47817/154454)
Branch Coverage 27.43% (24482/89254)

@924060929
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10860/26443)
Line Coverage 31.88% (92716/290862)
Region Coverage 30.96% (47824/154454)
Branch Coverage 27.43% (24482/89254)

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39533 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d7f7629c490b96729081192d7bbb84262d467e3, data reload: false

------ Round 1 ----------------------------------
q1	17560	6733	6593	6593
q2	2085	168	160	160
q3	10636	1068	1149	1068
q4	10534	742	784	742
q5	7731	2825	2779	2779
q6	215	133	130	130
q7	957	616	602	602
q8	9391	1916	2052	1916
q9	6613	6409	6387	6387
q10	7009	2262	2300	2262
q11	476	256	261	256
q12	392	203	206	203
q13	17773	2951	2982	2951
q14	236	205	207	205
q15	498	462	457	457
q16	509	379	387	379
q17	967	621	549	549
q18	7242	6620	6664	6620
q19	1390	1122	942	942
q20	470	207	204	204
q21	3995	3138	3311	3138
q22	1134	1012	990	990
Total cold run time: 107813 ms
Total hot run time: 39533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6562	6557	6485	6485
q2	330	233	233	233
q3	2836	2738	2812	2738
q4	2043	1767	1787	1767
q5	5795	5695	5692	5692
q6	208	131	128	128
q7	2195	1822	1789	1789
q8	3385	3503	3538	3503
q9	8867	8756	8911	8756
q10	3527	3532	3507	3507
q11	598	498	492	492
q12	815	592	592	592
q13	9255	3117	3231	3117
q14	291	273	278	273
q15	506	463	466	463
q16	510	441	440	440
q17	1825	1624	1585	1585
q18	8360	7726	7658	7658
q19	1686	1505	1603	1505
q20	2154	1825	1828	1825
q21	5399	5211	5162	5162
q22	1110	1036	1031	1031
Total cold run time: 68257 ms
Total hot run time: 58741 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10860/26443)
Line Coverage 31.88% (92736/290883)
Region Coverage 30.96% (47821/154473)
Branch Coverage 27.43% (24484/89260)

@doris-robot
Copy link

TPC-DS: Total hot run time: 197150 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d7f7629c490b96729081192d7bbb84262d467e3, data reload: false

query1	1295	910	889	889
query2	6310	1930	1866	1866
query3	10867	4221	4182	4182
query4	61535	29554	23584	23584
query5	5203	467	448	448
query6	410	173	167	167
query7	5486	318	299	299
query8	321	236	230	230
query9	8495	2583	2584	2583
query10	469	264	252	252
query11	17627	15193	15701	15193
query12	159	112	109	109
query13	1451	447	450	447
query14	9758	7126	7099	7099
query15	202	181	180	180
query16	7218	510	500	500
query17	1139	585	599	585
query18	1884	329	354	329
query19	211	165	154	154
query20	116	107	109	107
query21	206	105	107	105
query22	4728	4524	4431	4431
query23	34811	34637	34467	34467
query24	6148	2860	2959	2860
query25	531	416	416	416
query26	650	171	165	165
query27	2030	365	354	354
query28	4265	2140	2114	2114
query29	744	479	453	453
query30	232	160	153	153
query31	981	813	804	804
query32	63	59	57	57
query33	421	285	276	276
query34	910	507	506	506
query35	865	726	745	726
query36	1073	957	916	916
query37	119	70	75	70
query38	3980	3933	3985	3933
query39	1530	1491	1490	1490
query40	204	102	99	99
query41	51	47	55	47
query42	118	103	107	103
query43	523	493	474	474
query44	1174	802	806	802
query45	189	174	180	174
query46	1148	725	722	722
query47	2024	1946	1951	1946
query48	481	382	384	382
query49	717	402	415	402
query50	856	437	425	425
query51	7489	7216	7152	7152
query52	99	85	89	85
query53	253	177	182	177
query54	564	485	476	476
query55	77	75	80	75
query56	256	251	286	251
query57	1268	1190	1150	1150
query58	219	213	229	213
query59	3216	3026	3093	3026
query60	289	260	256	256
query61	111	104	108	104
query62	783	693	679	679
query63	213	191	186	186
query64	1412	668	621	621
query65	3342	3245	3183	3183
query66	711	292	292	292
query67	16039	15609	15657	15609
query68	4116	584	573	573
query69	437	269	263	263
query70	1147	1084	1144	1084
query71	352	249	248	248
query72	6140	4149	3980	3980
query73	739	337	359	337
query74	10178	9209	9357	9209
query75	3359	2634	2620	2620
query76	2200	1068	1036	1036
query77	465	267	264	264
query78	10657	9601	9597	9597
query79	1415	606	587	587
query80	953	417	414	414
query81	527	219	214	214
query82	175	91	86	86
query83	168	147	154	147
query84	282	75	72	72
query85	963	299	303	299
query86	371	294	291	291
query87	4376	4209	4195	4195
query88	4179	2383	2373	2373
query89	404	285	288	285
query90	1971	182	183	182
query91	183	144	149	144
query92	62	49	47	47
query93	2222	557	555	555
query94	748	299	296	296
query95	352	259	250	250
query96	611	283	277	277
query97	3308	3146	3180	3146
query98	211	203	195	195
query99	1572	1286	1274	1274
Total cold run time: 313112 ms
Total hot run time: 197150 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d7f7629c490b96729081192d7bbb84262d467e3, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.53	0.53	0.51
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.52	0.50
query10	0.54	0.55	0.55
query11	0.14	0.11	0.10
query12	0.13	0.10	0.11
query13	0.60	0.60	0.59
query14	0.79	0.77	0.82
query15	0.85	0.82	0.83
query16	0.38	0.38	0.38
query17	1.02	0.99	1.00
query18	0.24	0.23	0.22
query19	1.96	1.81	1.87
query20	0.01	0.01	0.02
query21	15.40	0.58	0.57
query22	2.50	2.32	2.06
query23	17.02	0.91	0.83
query24	3.06	0.90	0.99
query25	0.20	0.10	0.13
query26	0.43	0.13	0.13
query27	0.04	0.04	0.04
query28	10.59	0.51	0.45
query29	12.57	3.24	3.22
query30	0.25	0.06	0.07
query31	2.85	0.41	0.38
query32	3.27	0.46	0.45
query33	2.98	2.99	3.01
query34	16.98	4.45	4.48
query35	4.55	4.53	4.53
query36	0.65	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.73 s
Total hot run time: 30.26 s

@924060929
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10861/26443)
Line Coverage 31.87% (92705/290884)
Region Coverage 30.96% (47824/154478)
Branch Coverage 27.43% (24486/89264)

dataroaring
dataroaring approved these changes May 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6e38b0a into apache:branch-3.0 May 27, 2025
20 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@924060929 @hello-stephen @doris-robot @dataroaring @HappenLee