Skip to content

branch-3.1: [fix](nereids) Fix query err when topn push through union_all #54158 #54260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
github-actions wants to merge 1 commit into from
Closed

branch-3.1: [fix](nereids) Fix query err when topn push through union_all #54158 #54260

github-actions wants to merge 1 commit into from

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 4, 2025

Cherry-picked from #54158

@seawinde
[fix](nereids) Fix query err when topn push through union_all (#54158)
c35d3d3 
### What problem does this PR solve?

Related PR: #27535 

Problem Summary:

Fix query err when topn push through union_all
such veiw def and query are as following:

CREATE VIEW `view_test`AS 
SELECT   t1.`logTimestamp`,
         t1.`args1`,
         t1.`args2`,
         t1.`args3`,
         t1.`args4`,
         t1.`args5`,
         t1.`args6`,
         t1.`args7`,
         t1.`args8`,
         t1.`args9`,
         t1.`args10`,
         t1.`args11`,
         t1.`args12`,
         t1.`args13`,
         t1.`args14`,
         t1.`args15`,
         t1.`log`
FROM t1
UNION all
SELECT   t2.`logTimestamp`,
         t2.`args1`,
         t2.`args2`,
         t2.`args3`,
         t2.`args4`,
         t2.`args5`,
         t2.`args6`,
         t2.`args7`,
         t2.`args8`,
         t2.`args9`,
         t2.`args10`,
         t2.`args11`,
         t2.`args12`,
         t2.`args13`,
         t2.`args14`,
         t2.`args15`,
         t2.`log`
FROM t2; 

SELECT `args1`, `args2`, `args3`, `args4`, `args5`, `args6`, `args7`, `args8`, `args9`, `args10`, `args11`, `args12`, `args13`, `args14`, `args15`, `log`, logTimestamp
FROM view_test
ORDER BY logTimestamp desc
LIMIT 8;

would through exception as following, because topN use wrong ordery key
`log`
`Doris hll, bitmap, array, map, struct, jsonb, variant column must use
with specific function, and don't support filter, group by or order by.
please run 'help hll' or 'help bitmap' or 'help array' or 'help map' or
'help struct' or 'help jsonb' or 'help variant' in your mysql client.`
the pr fix this.
@github-actions github-actions bot requested a review from morrySnow as a code owner August 4, 2025 03:03
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Aug 4, 2025
@dataroaring dataroaring reopened this Aug 4, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-H: Total hot run time: 32234 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c35d3d31ccf360290bb3d423ca2027ae326634cb, data reload: false

------ Round 1 ----------------------------------
q1	17593	5473	5354	5354
q2	2063	287	162	162
q3	10453	1225	707	707
q4	10227	858	446	446
q5	8217	2371	2091	2091
q6	175	163	129	129
q7	894	748	605	605
q8	9326	1420	1186	1186
q9	5151	4895	4948	4895
q10	6742	2252	1797	1797
q11	480	272	269	269
q12	329	353	205	205
q13	17772	3613	3022	3022
q14	236	222	206	206
q15	525	474	473	473
q16	423	426	367	367
q17	570	864	367	367
q18	6986	6339	6330	6330
q19	1451	957	536	536
q20	319	352	205	205
q21	2797	2141	1923	1923
q22	1041	1055	959	959
Total cold run time: 103770 ms
Total hot run time: 32234 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5556	5518	5482	5482
q2	230	330	237	237
q3	2207	2598	2289	2289
q4	1337	1767	1363	1363
q5	4380	4753	4889	4753
q6	176	167	127	127
q7	2003	1927	1778	1778
q8	2640	2789	2667	2667
q9	7268	7113	7228	7113
q10	3044	3272	2836	2836
q11	568	513	493	493
q12	626	722	582	582
q13	3297	3786	3141	3141
q14	284	299	284	284
q15	525	467	476	467
q16	437	496	429	429
q17	1210	1743	1265	1265
q18	7582	7407	7385	7385
q19	756	1159	1030	1030
q20	1995	2043	1887	1887
q21	5239	4849	4682	4682
q22	1085	1098	997	997
Total cold run time: 52445 ms
Total hot run time: 51287 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-DS: Total hot run time: 196524 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c35d3d31ccf360290bb3d423ca2027ae326634cb, data reload: false

query1	1315	962	895	895
query2	6229	1956	1963	1956
query3	10915	4421	4598	4421
query4	33369	24025	23849	23849
query5	4326	599	468	468
query6	297	192	183	183
query7	3980	485	315	315
query8	310	253	261	253
query9	9463	2558	2542	2542
query10	483	326	246	246
query11	17971	15598	15170	15170
query12	154	109	99	99
query13	1563	517	385	385
query14	8987	6489	6662	6489
query15	218	209	191	191
query16	7786	588	450	450
query17	1604	770	585	585
query18	1992	419	326	326
query19	200	196	164	164
query20	134	116	118	116
query21	205	127	110	110
query22	4613	4702	4450	4450
query23	35131	34539	33974	33974
query24	8571	2673	2655	2655
query25	517	500	448	448
query26	1087	291	172	172
query27	2770	484	355	355
query28	5721	2192	2160	2160
query29	652	609	488	488
query30	239	188	166	166
query31	984	927	846	846
query32	81	62	61	61
query33	506	377	323	323
query34	734	847	533	533
query35	773	814	706	706
query36	991	1059	960	960
query37	110	92	67	67
query38	3980	3977	3956	3956
query39	1516	1475	1479	1475
query40	205	113	103	103
query41	47	52	48	48
query42	120	102	98	98
query43	500	514	487	487
query44	1315	814	836	814
query45	187	178	168	168
query46	880	1063	664	664
query47	2007	2014	1933	1933
query48	391	436	351	351
query49	721	489	396	396
query50	665	690	433	433
query51	7402	7290	7324	7290
query52	109	103	94	94
query53	224	258	204	204
query54	538	537	465	465
query55	77	76	81	76
query56	270	284	261	261
query57	1277	1271	1233	1233
query58	242	226	247	226
query59	3078	3236	3027	3027
query60	295	283	269	269
query61	141	134	127	127
query62	788	804	682	682
query63	229	201	199	199
query64	3171	1001	657	657
query65	3359	3305	3286	3286
query66	1005	413	314	314
query67	16014	15823	15631	15631
query68	7470	828	536	536
query69	521	299	260	260
query70	1159	1158	1104	1104
query71	382	313	261	261
query72	5726	3729	3857	3729
query73	643	738	351	351
query74	10569	9319	8928	8928
query75	3170	3149	2663	2663
query76	3098	1176	759	759
query77	476	371	270	270
query78	10258	10603	9685	9685
query79	2579	884	609	609
query80	615	517	422	422
query81	510	266	218	218
query82	431	114	89	89
query83	159	159	142	142
query84	250	99	83	83
query85	783	381	296	296
query86	406	300	309	300
query87	4299	4277	4212	4212
query88	4803	2384	2366	2366
query89	393	333	296	296
query90	1771	187	189	187
query91	138	140	112	112
query92	65	55	51	51
query93	2036	869	541	541
query94	708	394	321	321
query95	344	271	269	269
query96	478	599	281	281
query97	3254	3242	3126	3126
query98	221	211	201	201
query99	2310	1397	1312	1312
Total cold run time: 295759 ms
Total hot run time: 196524 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

ClickBench: Total hot run time: 29.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c35d3d31ccf360290bb3d423ca2027ae326634cb, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.56	0.50	0.51
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.51
query10	0.57	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.60	0.60
query14	0.78	0.79	0.79
query15	0.85	0.83	0.82
query16	0.38	0.39	0.39
query17	1.09	1.00	1.01
query18	0.23	0.22	0.21
query19	1.82	1.86	1.85
query20	0.02	0.01	0.02
query21	15.38	0.93	0.57
query22	0.74	0.78	0.70
query23	15.09	1.48	0.56
query24	2.85	1.58	2.33
query25	0.26	0.25	0.05
query26	0.16	0.13	0.14
query27	0.05	0.06	0.06
query28	14.34	1.00	0.44
query29	12.58	3.89	3.27
query30	0.24	0.09	0.07
query31	2.81	0.59	0.38
query32	3.22	0.54	0.46
query33	2.96	3.00	3.00
query34	16.70	5.15	4.57
query35	4.58	4.55	4.52
query36	0.62	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.99 s
Total hot run time: 29.56 s

@morrySnow morrySnow closed this Aug 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @dataroaring @morrySnow @seawinde