Skip to content

[fix](nereids)Fix decompose repeat nest rewrite need to derive stats after construct cteProducer#60811

Open
feiniaofeiafei wants to merge 3 commits intoapache:masterfrom
feiniaofeiafei:fix-DecomposeRepeatWithPreAggregation-with-nest-rewrite
Open

[fix](nereids)Fix decompose repeat nest rewrite need to derive stats after construct cteProducer#60811
feiniaofeiafei wants to merge 3 commits intoapache:masterfrom
feiniaofeiafei:fix-DecomposeRepeatWithPreAggregation-with-nest-rewrite

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Feb 24, 2026

What problem does this PR solve?

Reproduction SQL:

    select a,b,c,c1 from (
    select a,b,c,d,sum(d) c1 from t1 group by grouping sets((a,b,c),(a,b,c,d),(a),(a,b,c,c))
    ) t group by rollup(a,b,c,c1);

Error:

java.lang.IllegalArgumentException: Stats for CTE: CTEId#0 not found

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Root cause:
This SQL triggers DecomposeRepeatWithPreAggregation twice: first for the inner LogicalAggregate, then for the outer LogicalAggregate.
When rewriting the inner aggregate, a CTE structure is created (CTEAnchor, CTEProducer, CTEConsumer).
When rewriting the outer aggregate, choosePreAggShuffleKeyPartitionExprs calls StatsDerive to derive statistics for the current subtree.
At that point, the child of the outer LogicalAggregate is only the consumer subtree; the producer subtree is not part of the plan being visited.
StatsDerive.visitLogicalCTEConsumer looks up producer statistics by CTEId, but the producer was never derived in this context, so the lookup fails with Stats for CTE: CTEId#0 not found.
How to fix:
In DecomposeRepeatWithPreAggregation.constructProducer(), derive statistics for the CTE producer when it is created. This ensures producer stats are stored in the context before the consumer is later visited. When StatsDerive visits LogicalCTEConsumer during the outer rewrite, it can find the producer stats correctly.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 24, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28882 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 646284915155304b709f9b829a9a62f01e120aac, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17666	4469	4324	4324
q2	q3	10646	781	522	522
q4	4685	352	249	249
q5	7557	1211	1052	1052
q6	180	178	149	149
q7	773	835	671	671
q8	9443	1467	1322	1322
q9	4812	4756	4714	4714
q10	6847	1892	1641	1641
q11	528	255	238	238
q12	750	566	476	476
q13	17836	4292	3450	3450
q14	236	238	215	215
q15	990	819	812	812
q16	785	735	686	686
q17	732	882	421	421
q18	6063	5353	5303	5303
q19	1205	974	594	594
q20	502	493	398	398
q21	4497	1846	1404	1404
q22	336	290	241	241
Total cold run time: 97069 ms
Total hot run time: 28882 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4421	4373	4390	4373
q2	q3	1764	2180	1732	1732
q4	836	1155	736	736
q5	4012	4302	4319	4302
q6	180	169	142	142
q7	1741	1591	1481	1481
q8	2399	2625	2467	2467
q9	7457	7447	7419	7419
q10	2610	2850	2422	2422
q11	503	450	430	430
q12	509	620	463	463
q13	4039	4453	3647	3647
q14	336	302	289	289
q15	855	804	813	804
q16	707	758	708	708
q17	1187	1533	1307	1307
q18	7049	6851	6626	6626
q19	910	867	874	867
q20	2102	2183	2003	2003
q21	4087	3541	3852	3541
q22	477	467	413	413
Total cold run time: 48181 ms
Total hot run time: 46172 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184265 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 646284915155304b709f9b829a9a62f01e120aac, data reload: false

query5	5346	646	534	534
query6	334	213	218	213
query7	4226	475	278	278
query8	345	245	244	244
query9	8794	2755	2686	2686
query10	582	379	333	333
query11	17034	17459	17197	17197
query12	227	140	143	140
query13	1486	523	361	361
query14	7330	3418	3301	3301
query14_1	2949	2997	2937	2937
query15	222	200	171	171
query16	1022	449	480	449
query17	1463	755	630	630
query18	2856	456	360	360
query19	245	214	177	177
query20	140	137	135	135
query21	218	150	137	137
query22	5498	5098	4725	4725
query23	17252	16756	16645	16645
query23_1	16812	16842	16747	16747
query24	7178	1611	1222	1222
query24_1	1232	1240	1241	1240
query25	554	479	446	446
query26	1246	260	154	154
query27	2770	476	289	289
query28	4492	1846	1849	1846
query29	822	576	488	488
query30	308	246	220	220
query31	885	730	655	655
query32	81	73	75	73
query33	549	340	303	303
query34	911	909	571	571
query35	649	674	602	602
query36	1099	1164	926	926
query37	130	112	86	86
query38	2937	2903	2865	2865
query39	952	864	822	822
query39_1	820	823	851	823
query40	235	157	151	151
query41	68	64	69	64
query42	108	104	103	103
query43	385	388	354	354
query44	
query45	198	194	187	187
query46	882	989	623	623
query47	2118	2124	2042	2042
query48	317	319	241	241
query49	644	476	419	419
query50	684	277	219	219
query51	4097	4113	4037	4037
query52	109	110	96	96
query53	293	348	285	285
query54	307	270	309	270
query55	91	85	78	78
query56	307	305	345	305
query57	1390	1340	1255	1255
query58	291	272	274	272
query59	2608	2655	2573	2573
query60	335	337	328	328
query61	152	153	151	151
query62	618	583	558	558
query63	309	279	282	279
query64	4815	1286	1006	1006
query65	
query66	1369	452	369	369
query67	16475	16409	16359	16359
query68	
query69	394	354	291	291
query70	969	974	947	947
query71	333	309	297	297
query72	2926	2707	2412	2412
query73	534	540	319	319
query74	9985	9888	9753	9753
query75	2844	2747	2465	2465
query76	2293	1043	679	679
query77	379	386	311	311
query78	11174	11468	10692	10692
query79	1140	845	584	584
query80	1414	623	543	543
query81	548	280	266	266
query82	1001	147	114	114
query83	339	260	246	246
query84	252	125	102	102
query85	922	495	437	437
query86	447	307	301	301
query87	3112	3145	3010	3010
query88	3554	2669	2654	2654
query89	423	372	368	368
query90	1949	172	168	168
query91	167	159	136	136
query92	79	78	73	73
query93	925	861	501	501
query94	651	281	285	281
query95	568	416	315	315
query96	631	519	228	228
query97	2513	2493	2435	2435
query98	227	215	230	215
query99	1042	1003	920	920
Total cold run time: 256212 ms
Total hot run time: 184265 ms

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@feiniaofeiafei
Copy link
Contributor Author

run cloud_p0

@feiniaofeiafei
Copy link
Contributor Author

run p0

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@feiniaofeiafei
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28879 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29add004c23521da24a2144493f13b03dc53ad42, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17649	4442	4312	4312
q2	q3	10658	781	531	531
q4	4686	354	251	251
q5	7549	1181	1014	1014
q6	174	177	149	149
q7	768	850	658	658
q8	9294	1453	1281	1281
q9	4846	4828	4741	4741
q10	6758	1865	1642	1642
q11	440	264	254	254
q12	725	559	454	454
q13	17765	4218	3397	3397
q14	226	229	216	216
q15	878	790	783	783
q16	732	707	668	668
q17	724	834	419	419
q18	5928	5376	5303	5303
q19	1244	970	606	606
q20	486	487	381	381
q21	4919	2003	1522	1522
q22	348	304	297	297
Total cold run time: 96797 ms
Total hot run time: 28879 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4693	4495	4531	4495
q2	q3	1845	2204	1790	1790
q4	846	1198	749	749
q5	4024	4398	4352	4352
q6	186	169	136	136
q7	1766	1612	1666	1612
q8	2492	2633	2624	2624
q9	7639	7468	7433	7433
q10	2647	2848	2504	2504
q11	524	443	433	433
q12	517	598	453	453
q13	3950	4425	3551	3551
q14	280	295	272	272
q15	879	814	796	796
q16	698	776	691	691
q17	1139	1518	1302	1302
q18	7021	6649	6725	6649
q19	866	864	831	831
q20	2077	2213	2055	2055
q21	3917	3428	3324	3324
q22	433	436	381	381
Total cold run time: 48439 ms
Total hot run time: 46433 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184131 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29add004c23521da24a2144493f13b03dc53ad42, data reload: false

query5	5441	663	549	549
query6	327	229	211	211
query7	4228	455	260	260
query8	327	238	241	238
query9	8750	2786	2732	2732
query10	579	415	365	365
query11	17039	17571	17444	17444
query12	202	134	124	124
query13	1334	495	355	355
query14	7847	3297	3163	3163
query14_1	2907	2926	2938	2926
query15	217	221	178	178
query16	1056	516	464	464
query17	1442	716	578	578
query18	2703	433	337	337
query19	214	213	175	175
query20	134	129	129	129
query21	210	135	108	108
query22	4817	4878	4764	4764
query23	17129	16781	16603	16603
query23_1	16606	16663	16612	16612
query24	6837	1577	1246	1246
query24_1	1250	1217	1233	1217
query25	546	463	416	416
query26	1233	260	147	147
query27	2783	466	282	282
query28	4530	1861	1873	1861
query29	803	561	496	496
query30	318	245	212	212
query31	848	745	660	660
query32	84	79	75	75
query33	533	345	298	298
query34	897	914	561	561
query35	641	680	594	594
query36	1070	1104	1003	1003
query37	141	103	82	82
query38	2927	2918	2846	2846
query39	906	847	831	831
query39_1	863	833	822	822
query40	234	153	140	140
query41	69	64	65	64
query42	120	109	111	109
query43	391	403	354	354
query44	
query45	201	190	185	185
query46	881	992	597	597
query47	2118	2152	2065	2065
query48	316	323	239	239
query49	651	475	389	389
query50	688	290	221	221
query51	4068	4118	4045	4045
query52	111	110	109	109
query53	294	350	288	288
query54	311	282	275	275
query55	91	90	85	85
query56	342	315	324	315
query57	1368	1326	1265	1265
query58	322	286	282	282
query59	2594	2685	2581	2581
query60	352	361	345	345
query61	176	193	143	143
query62	617	590	539	539
query63	320	274	273	273
query64	4869	1255	986	986
query65	
query66	1370	453	356	356
query67	16444	16370	16364	16364
query68	
query69	385	300	275	275
query70	997	1009	965	965
query71	342	308	304	304
query72	2750	2623	2317	2317
query73	547	551	320	320
query74	9997	9951	9788	9788
query75	2841	2742	2463	2463
query76	2289	1042	674	674
query77	368	402	301	301
query78	11192	11485	10744	10744
query79	1154	786	603	603
query80	1392	632	541	541
query81	568	276	249	249
query82	1008	151	118	118
query83	330	259	246	246
query84	250	118	97	97
query85	897	482	423	423
query86	445	288	329	288
query87	3129	3143	3000	3000
query88	3547	2681	2652	2652
query89	420	366	341	341
query90	1964	182	179	179
query91	166	165	139	139
query92	81	80	75	75
query93	998	866	512	512
query94	644	328	290	290
query95	583	409	320	320
query96	657	534	233	233
query97	2454	2524	2416	2416
query98	227	216	221	216
query99	1012	999	917	917
Total cold run time: 255104 ms
Total hot run time: 184131 ms

@feiniaofeiafei
Copy link
Contributor Author

run external

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants