[bug](expr)fix operator do_projections should use local_state intermediate_projections #38612

zhangstar333 · 2024-07-31T10:59:09Z

should use local_state->_intermediate_projections, this is clone from parent intermediate_projections.
So it can run independently between different instance operators without affecting each other。
It needs to run in some extreme multi-threaded environments, so it's not easy to add test cases.

doris-robot · 2024-07-31T10:59:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zhangstar333 · 2024-07-31T10:59:18Z

run buildall

github-actions · 2024-07-31T11:05:45Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-31T11:50:52Z

TPC-H: Total hot run time: 41647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f061afd0f3380f4aaa14865c8850c0375e6ad932, data reload: false

------ Round 1 ----------------------------------
q1	17686	4140	4087	4087
q2	2044	208	206	206
q3	10538	1339	1336	1336
q4	10208	851	969	851
q5	7671	3006	2990	2990
q6	225	136	139	136
q7	1042	621	606	606
q8	9449	1759	1995	1759
q9	8673	6641	6618	6618
q10	8685	3833	3843	3833
q11	419	246	251	246
q12	407	227	223	223
q13	17768	2904	2935	2904
q14	270	242	241	241
q15	521	486	475	475
q16	489	391	383	383
q17	996	963	919	919
q18	8061	7362	7251	7251
q19	1377	1219	1223	1219
q20	580	332	330	330
q21	5349	4749	4804	4749
q22	356	305	285	285
Total cold run time: 112814 ms
Total hot run time: 41647 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4033	3986	3987	3986
q2	328	227	227	227
q3	3001	3013	3175	3013
q4	2034	2046	1946	1946
q5	5638	5493	5439	5439
q6	220	133	131	131
q7	2136	1804	1830	1804
q8	3336	3428	3368	3368
q9	8737	8730	8855	8730
q10	3932	4086	3983	3983
q11	549	462	467	462
q12	790	617	588	588
q13	16118	3142	3151	3142
q14	292	288	273	273
q15	532	496	475	475
q16	476	402	416	402
q17	1795	1744	1706	1706
q18	8360	7837	7765	7765
q19	1742	1724	1729	1724
q20	2064	1852	1829	1829
q21	5716	5353	5248	5248
q22	526	476	466	466
Total cold run time: 72355 ms
Total hot run time: 56707 ms

HappenLee · 2024-07-31T11:51:51Z

Add TEST and DESC of the PR

doris-robot · 2024-07-31T12:01:31Z

TPC-DS: Total hot run time: 170330 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f061afd0f3380f4aaa14865c8850c0375e6ad932, data reload: false

query1	908	377	370	370
query2	6462	1655	1653	1653
query3	6674	210	221	210
query4	20134	17292	17340	17292
query5	3655	515	516	515
query6	286	186	179	179
query7	4595	293	301	293
query8	278	208	192	192
query9	8509	2447	2415	2415
query10	477	288	271	271
query11	10379	10145	9906	9906
query12	116	88	83	83
query13	1628	386	369	369
query14	8764	8498	8347	8347
query15	207	156	158	156
query16	6942	464	461	461
query17	938	571	535	535
query18	1903	292	297	292
query19	191	147	142	142
query20	88	89	85	85
query21	199	102	95	95
query22	4133	3978	3921	3921
query23	33863	33390	33508	33390
query24	10298	3119	3056	3056
query25	671	455	424	424
query26	1676	154	153	153
query27	2959	277	297	277
query28	7653	2052	2035	2035
query29	1343	444	422	422
query30	236	152	152	152
query31	943	785	788	785
query32	99	52	57	52
query33	675	356	316	316
query34	946	510	493	493
query35	871	777	763	763
query36	1029	885	893	885
query37	295	82	85	82
query38	2912	2751	2761	2751
query39	865	824	810	810
query40	257	118	114	114
query41	45	45	45	45
query42	127	104	133	104
query43	468	421	413	413
query44	1188	725	732	725
query45	220	174	180	174
query46	1091	829	766	766
query47	1804	1688	1702	1688
query48	371	287	293	287
query49	933	427	430	427
query50	903	428	430	428
query51	6775	6682	6764	6682
query52	99	88	91	88
query53	250	182	185	182
query54	604	454	459	454
query55	77	74	79	74
query56	282	247	250	247
query57	1138	1026	1039	1026
query58	269	284	280	280
query59	2474	2260	2373	2260
query60	302	272	263	263
query61	96	91	93	91
query62	874	677	657	657
query63	212	182	187	182
query64	5636	1912	1859	1859
query65	3135	3078	3104	3078
query66	1316	326	321	321
query67	15148	14816	14706	14706
query68	4220	553	577	553
query69	446	307	295	295
query70	1092	1087	1068	1068
query71	419	266	273	266
query72	7077	2675	2482	2482
query73	749	328	327	327
query74	6050	5560	5529	5529
query75	3359	2741	2736	2736
query76	2294	1350	1401	1350
query77	417	302	306	302
query78	9434	8883	8846	8846
query79	1603	539	546	539
query80	1212	513	495	495
query81	555	225	231	225
query82	1009	124	128	124
query83	260	172	177	172
query84	259	79	77	77
query85	1169	330	297	297
query86	397	292	300	292
query87	3258	3061	3034	3034
query88	2970	2505	2484	2484
query89	396	290	286	286
query90	1689	190	185	185
query91	129	100	99	99
query92	57	49	50	49
query93	1447	618	608	608
query94	767	292	315	292
query95	381	285	272	272
query96	588	278	280	278
query97	3225	3024	3026	3024
query98	221	202	203	202
query99	1671	1323	1293	1293
Total cold run time: 259714 ms
Total hot run time: 170330 ms

doris-robot · 2024-07-31T12:07:01Z

ClickBench: Total hot run time: 29.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f061afd0f3380f4aaa14865c8850c0375e6ad932, data reload: false

query1	0.05	0.03	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.06	0.06
query5	0.48	0.48	0.48
query6	1.14	0.71	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.59	0.51	0.50
query10	0.55	0.56	0.57
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.76	0.79	0.80
query15	0.90	0.85	0.85
query16	0.36	0.35	0.35
query17	1.00	0.95	1.00
query18	0.22	0.21	0.20
query19	1.81	1.70	1.73
query20	0.01	0.02	0.02
query21	15.40	0.77	0.65
query22	4.61	8.21	1.06
query23	17.96	1.41	1.34
query24	2.31	0.23	0.22
query25	0.19	0.08	0.08
query26	0.32	0.22	0.21
query27	0.45	0.23	0.23
query28	13.22	1.00	0.96
query29	12.51	3.30	3.27
query30	0.26	0.05	0.05
query31	2.88	0.41	0.40
query32	3.23	0.48	0.48
query33	2.96	2.92	3.02
query34	15.45	4.25	4.24
query35	4.30	4.29	4.28
query36	0.67	0.48	0.49
query37	0.19	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 108.32 s
Total hot run time: 29.63 s

…ediate_projections

zhangstar333 · 2024-08-01T03:03:57Z

run buildall

github-actions · 2024-08-01T03:09:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-01T03:37:14Z

TPC-H: Total hot run time: 41875 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f01f12982419b32ad52bb9f3700c086fdbb4016d, data reload: false

------ Round 1 ----------------------------------
q1	18727	4120	4098	4098
q2	2030	198	191	191
q3	10483	1313	1307	1307
q4	11669	852	935	852
q5	7610	2964	2987	2964
q6	222	138	136	136
q7	1033	617	617	617
q8	9598	1921	1928	1921
q9	8764	6652	6721	6652
q10	8943	4020	4067	4020
q11	435	258	259	258
q12	470	232	231	231
q13	17925	2949	2956	2949
q14	269	243	247	243
q15	536	486	500	486
q16	555	400	406	400
q17	1311	923	867	867
q18	9592	7261	7215	7215
q19	1597	1204	1210	1204
q20	570	324	333	324
q21	5302	4663	4718	4663
q22	350	277	279	277
Total cold run time: 117991 ms
Total hot run time: 41875 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4108	4017	4039	4017
q2	329	221	218	218
q3	3137	3161	3176	3161
q4	1985	2012	1973	1973
q5	5462	5463	5419	5419
q6	228	133	134	133
q7	2194	1769	1785	1769
q8	3336	3393	3342	3342
q9	8636	8805	8690	8690
q10	3966	3862	4012	3862
q11	578	463	450	450
q12	759	617	582	582
q13	16342	3077	3096	3077
q14	304	279	264	264
q15	536	481	510	481
q16	465	439	434	434
q17	1765	1755	1740	1740
q18	8247	7934	7749	7749
q19	3318	1717	1716	1716
q20	2071	1843	1830	1830
q21	5733	5306	5399	5306
q22	534	459	465	459
Total cold run time: 74033 ms
Total hot run time: 56672 ms

doris-robot · 2024-08-01T03:48:02Z

TPC-DS: Total hot run time: 170239 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f01f12982419b32ad52bb9f3700c086fdbb4016d, data reload: false

query1	918	388	376	376
query2	7618	1754	1736	1736
query3	6950	209	220	209
query4	20396	17677	17245	17245
query5	3604	504	520	504
query6	289	179	178	178
query7	4601	294	295	294
query8	249	197	187	187
query9	8497	2405	2384	2384
query10	422	269	258	258
query11	10517	9948	9920	9920
query12	125	90	90	90
query13	1634	387	370	370
query14	9522	7843	7311	7311
query15	207	156	161	156
query16	6973	464	470	464
query17	937	571	540	540
query18	1910	283	288	283
query19	188	144	142	142
query20	91	86	87	86
query21	202	100	98	98
query22	4350	4221	4098	4098
query23	33735	33618	33636	33618
query24	10316	3084	3106	3084
query25	683	409	412	409
query26	1704	158	159	158
query27	3302	294	286	286
query28	7665	2031	2029	2029
query29	1359	475	449	449
query30	257	165	161	161
query31	970	774	769	769
query32	100	55	58	55
query33	698	326	340	326
query34	968	497	487	487
query35	864	774	743	743
query36	1067	894	857	857
query37	288	80	78	78
query38	2904	2790	2764	2764
query39	866	796	805	796
query40	256	109	111	109
query41	48	41	42	41
query42	121	97	94	94
query43	474	426	434	426
query44	1174	721	729	721
query45	210	180	187	180
query46	1092	828	785	785
query47	1792	1709	1700	1700
query48	366	298	292	292
query49	941	412	429	412
query50	925	432	440	432
query51	6777	6781	6590	6590
query52	99	89	88	88
query53	250	177	175	175
query54	665	452	449	449
query55	75	73	71	71
query56	277	257	250	250
query57	1157	1021	1032	1021
query58	265	267	278	267
query59	2851	2509	2506	2506
query60	300	273	268	268
query61	93	92	91	91
query62	893	666	671	666
query63	203	182	180	180
query64	5623	1923	1970	1923
query65	3161	3189	3086	3086
query66	1315	345	338	338
query67	15239	14775	14600	14600
query68	4280	570	596	570
query69	613	380	316	316
query70	1114	1053	1037	1037
query71	445	287	276	276
query72	7248	2812	2628	2628
query73	770	332	333	332
query74	6082	5649	5566	5566
query75	3375	2763	2742	2742
query76	2060	1368	1422	1368
query77	552	315	310	310
query78	9511	8877	8890	8877
query79	3522	526	527	526
query80	2029	518	519	518
query81	574	229	227	227
query82	477	137	136	136
query83	294	174	175	174
query84	269	84	89	84
query85	1224	361	485	361
query86	445	312	309	309
query87	3273	3123	3071	3071
query88	3039	2479	2465	2465
query89	392	302	299	299
query90	1660	196	191	191
query91	124	98	97	97
query92	60	47	50	47
query93	2286	616	617	616
query94	801	300	270	270
query95	441	267	266	266
query96	602	279	281	279
query97	3202	3040	3077	3040
query98	242	201	197	197
query99	1632	1308	1285	1285
Total cold run time: 267226 ms
Total hot run time: 170239 ms

doris-robot · 2024-08-01T03:53:32Z

ClickBench: Total hot run time: 30.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f01f12982419b32ad52bb9f3700c086fdbb4016d, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.06	0.06
query5	0.50	0.48	0.47
query6	1.13	0.71	0.71
query7	0.02	0.02	0.02
query8	0.06	0.04	0.04
query9	0.57	0.51	0.51
query10	0.56	0.57	0.56
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.60
query14	0.77	0.77	0.79
query15	0.90	0.85	0.85
query16	0.35	0.35	0.35
query17	1.01	1.03	1.03
query18	0.22	0.21	0.21
query19	1.90	1.70	1.75
query20	0.01	0.00	0.00
query21	15.40	0.78	0.67
query22	3.94	8.16	1.48
query23	17.99	1.27	1.24
query24	2.26	0.22	0.22
query25	0.18	0.08	0.08
query26	0.31	0.22	0.20
query27	0.45	0.23	0.23
query28	13.18	0.99	0.96
query29	12.60	3.29	3.24
query30	0.26	0.07	0.06
query31	2.86	0.40	0.39
query32	3.26	0.49	0.49
query33	2.90	2.98	2.98
query34	15.47	4.29	4.26
query35	4.33	4.30	4.30
query36	0.67	0.48	0.51
query37	0.18	0.15	0.16
query38	0.15	0.16	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.12
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.7 s
Total hot run time: 30.11 s

github-actions · 2024-08-02T02:45:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-02T02:45:40Z

PR approved by anyone and no changes requested.

HappenLee

LGTM

…diate_projections (apache#38612) should use local_state->_intermediate_projections, this is clone from parent intermediate_projections. So it can run independently between different instance operators without affecting each other。 It needs to run in some extreme multi-threaded environments, so it's not easy to add test cases.

…diate_projections (#38612) should use local_state->_intermediate_projections, this is clone from parent intermediate_projections. So it can run independently between different instance operators without affecting each other。 It needs to run in some extreme multi-threaded environments, so it's not easy to add test cases.

…t null) case (#38756) ## Proposed changes Issue Number: close #38612

…state intermediate_projections (#38612) (#38765) ## Proposed changes cherry-pick from master #38612

gavinchou added the dev/3.0.x label Jul 31, 2024

zhangstar333 changed the title ~~[bug](expr)fix operator do_projections without use local_state intermediate_projections~~ [bug](expr)fix operator do_projections should use local_state intermediate_projections Jul 31, 2024

yiguolei added p0_c dev/2.1.x labels Jul 31, 2024

zhangstar333 added 2 commits August 1, 2024 10:00

[bug](expr)fix operator do_projections without use local_state interm…

2ebd3ac

…ediate_projections

update

f01f129

zhangstar333 force-pushed the fix_udf_thread branch from f061afd to f01f129 Compare August 1, 2024 03:03

github-actions bot added the doing label Aug 1, 2024

dataroaring added the usercase Important user case type label label Aug 2, 2024

yiguolei approved these changes Aug 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2024

github-actions bot added the reviewed label Aug 2, 2024

HappenLee approved these changes Aug 2, 2024

View reviewed changes

yiguolei merged commit 355a7d7 into apache:master Aug 2, 2024
28 of 30 checks passed

HappenLee mentioned this pull request Aug 2, 2024

[Bug](fix) fix coredump case in (not null, null) execpt (not null, not null) case #38756

Merged

zhangstar333 mentioned this pull request Aug 2, 2024

[cherry-pick](branch-21)fix operator do_projections should use local_state intermediate_projections (#38612) #38765

Merged

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Aug 2, 2024

dataroaring pushed a commit that referenced this pull request Aug 4, 2024

[Bug](fix) fix coredump case in (not null, null) execpt (not null, no…

7bdc508

…t null) case (#38756) ## Proposed changes Issue Number: close #38612

dataroaring added dev/2.1.6-merged and removed dev/2.1.x labels Aug 4, 2024

yiguolei pushed a commit that referenced this pull request Aug 5, 2024

[cherry-pick](branch-21)fix operator do_projections should use local_…

1b3d4b4

…state intermediate_projections (#38612) (#38765) ## Proposed changes cherry-pick from master #38612

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Open

yiguolei mentioned this pull request Sep 5, 2024

Release Note 2.1.6 #40443

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug](expr)fix operator do_projections should use local_state intermediate_projections #38612

[bug](expr)fix operator do_projections should use local_state intermediate_projections #38612

zhangstar333 commented Jul 31, 2024 •

edited

Loading

doris-robot commented Jul 31, 2024

zhangstar333 commented Jul 31, 2024

github-actions bot commented Jul 31, 2024

doris-robot commented Jul 31, 2024

HappenLee commented Jul 31, 2024

doris-robot commented Jul 31, 2024

doris-robot commented Jul 31, 2024

zhangstar333 commented Aug 1, 2024

github-actions bot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

github-actions bot commented Aug 2, 2024

github-actions bot commented Aug 2, 2024

HappenLee left a comment

[bug](expr)fix operator do_projections should use local_state intermediate_projections #38612

[bug](expr)fix operator do_projections should use local_state intermediate_projections #38612

Conversation

zhangstar333 commented Jul 31, 2024 • edited Loading

doris-robot commented Jul 31, 2024

zhangstar333 commented Jul 31, 2024

github-actions bot commented Jul 31, 2024

doris-robot commented Jul 31, 2024

HappenLee commented Jul 31, 2024

doris-robot commented Jul 31, 2024

doris-robot commented Jul 31, 2024

zhangstar333 commented Aug 1, 2024

github-actions bot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

doris-robot commented Aug 1, 2024

github-actions bot commented Aug 2, 2024

github-actions bot commented Aug 2, 2024

HappenLee left a comment

Choose a reason for hiding this comment

zhangstar333 commented Jul 31, 2024 •

edited

Loading