Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids): fix be core when array_map is not nullable (#24488) #38550

Merged
merged 4 commits into from
Aug 1, 2024
Merged

[fix](Nereids): fix be core when array_map is not nullable (#24488) #38550

merged 4 commits into from
Aug 1, 2024

Conversation

keanji-x
Copy link
Contributor

Proposed changes

#24488

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added area/nereids area/planner Issues or PRs related to the query planner kind/test labels Jul 31, 2024
@keanji-x
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21472)
Line Coverage: 29.47% (66580/225912)
Region Coverage: 28.96% (34327/118529)
Branch Coverage: 24.84% (17638/71014)
Coverage Report: http://coverage.selectdb-in.cc/coverage/727b71671100f543953b36f4be5e9554f33decf2_727b71671100f543953b36f4be5e9554f33decf2/report/index.html

@keanji-x
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@keanji-x keanji-x force-pushed the pick_88adab3 branch 3 times, most recently from 2954a9b to b2afc0a Compare July 31, 2024 05:52
@doris-robot
Copy link

TPC-H: Total hot run time: 49444 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3fee58c448c05be7b8b3e4921b554c42c4e5d4d2, data reload: false

------ Round 1 ----------------------------------
q1	17590	4382	4364	4364
q2	2066	152	154	152
q3	10445	1879	1948	1879
q4	10331	1237	1319	1237
q5	8769	3857	3916	3857
q6	267	125	126	125
q7	2049	1610	1620	1610
q8	9256	2697	2683	2683
q9	10591	10342	10064	10064
q10	8632	3469	3515	3469
q11	419	250	254	250
q12	476	296	307	296
q13	18353	3949	4034	3949
q14	355	334	331	331
q15	511	454	458	454
q16	672	572	576	572
q17	1126	948	927	927
q18	7360	6798	6857	6798
q19	1785	1662	1632	1632
q20	531	307	300	300
q21	4454	4113	4052	4052
q22	545	477	443	443
Total cold run time: 116583 ms
Total hot run time: 49444 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4332	4336	4343	4336
q2	328	224	223	223
q3	4191	4113	4135	4113
q4	2739	2728	2725	2725
q5	7092	7055	7003	7003
q6	235	119	120	119
q7	3223	2844	2892	2844
q8	4336	4428	4455	4428
q9	16993	16682	16694	16682
q10	4215	4247	4262	4247
q11	745	681	686	681
q12	1014	865	833	833
q13	6681	3768	3742	3742
q14	457	422	418	418
q15	496	451	461	451
q16	739	692	686	686
q17	3829	3867	3780	3780
q18	8740	8780	9666	8780
q19	1744	1724	1666	1666
q20	2420	2157	2097	2097
q21	8443	8445	8431	8431
q22	1062	967	1006	967
Total cold run time: 84054 ms
Total hot run time: 79252 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21472)
Line Coverage: 29.47% (66584/225912)
Region Coverage: 28.96% (34325/118529)
Branch Coverage: 24.83% (17635/71014)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3fee58c448c05be7b8b3e4921b554c42c4e5d4d2_3fee58c448c05be7b8b3e4921b554c42c4e5d4d2/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

4 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-DS: Total hot run time: 205083 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3fee58c448c05be7b8b3e4921b554c42c4e5d4d2, data reload: false

query1	938	421	379	379
query2	6555	2836	2762	2762
query3	6918	210	205	205
query4	20907	18123	18071	18071
query5	19751	6546	6533	6533
query6	295	229	240	229
query7	4160	307	336	307
query8	433	446	409	409
query9	3169	2734	2660	2660
query10	422	307	295	295
query11	11353	10784	10756	10756
query12	129	79	74	74
query13	5614	706	725	706
query14	17523	13413	13466	13413
query15	376	251	257	251
query16	6434	293	268	268
query17	1728	1453	879	879
query18	2304	418	414	414
query19	215	154	156	154
query20	81	80	81	80
query21	191	95	91	91
query22	5397	5031	5027	5027
query23	32854	32066	32160	32066
query24	6947	6514	6675	6514
query25	537	433	422	422
query26	536	171	167	167
query27	1873	310	301	301
query28	6216	2383	2351	2351
query29	2868	2800	2724	2724
query30	241	170	171	170
query31	904	747	744	744
query32	66	53	60	53
query33	407	255	256	255
query34	856	480	492	480
query35	1133	919	918	918
query36	1243	1136	1169	1136
query37	93	64	61	61
query38	3043	2969	2951	2951
query39	1391	1340	1341	1340
query40	210	99	105	99
query41	48	45	45	45
query42	91	92	95	92
query43	762	806	623	623
query44	1154	717	731	717
query45	255	238	241	238
query46	1229	995	1006	995
query47	1921	1761	1586	1586
query48	1029	722	726	722
query49	623	372	364	364
query50	869	616	602	602
query51	4766	4655	4783	4655
query52	99	83	80	80
query53	444	340	334	334
query54	2658	2450	2481	2450
query55	102	78	94	78
query56	229	230	217	217
query57	1177	1058	1146	1058
query58	222	211	212	211
query59	4011	3961	4234	3961
query60	221	205	212	205
query61	98	97	98	97
query62	751	454	470	454
query63	500	347	349	347
query64	2595	1541	1418	1418
query65	3669	3622	3605	3605
query66	773	381	388	381
query67	16556	17401	15843	15843
query68	7470	676	676	676
query69	574	362	355	355
query70	1602	1561	1375	1375
query71	409	323	316	316
query72	6530	3532	3497	3497
query73	734	319	333	319
query74	6324	5866	5821	5821
query75	4590	3731	3659	3659
query76	4594	1168	1170	1168
query77	556	270	254	254
query78	12442	11725	11977	11725
query79	8058	638	646	638
query80	2501	412	406	406
query81	531	239	232	232
query82	1515	100	97	97
query83	205	131	136	131
query84	262	73	72	72
query85	1430	339	334	334
query86	361	302	296	296
query87	3201	3032	3032	3032
query88	4979	2308	2312	2308
query89	381	292	284	284
query90	1756	206	211	206
query91	174	143	145	143
query92	61	53	56	53
query93	3537	615	596	596
query94	904	203	212	203
query95	1134	1068	1082	1068
query96	646	328	317	317
query97	6392	6314	6435	6314
query98	191	176	172	172
query99	2834	923	997	923
Total cold run time: 310041 ms
Total hot run time: 205083 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3fee58c448c05be7b8b3e4921b554c42c4e5d4d2, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.79	0.08	0.10
query5	0.54	0.52	0.52
query6	1.31	0.66	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.50	0.48
query10	0.54	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.78	0.80	0.77
query15	0.77	0.76	0.76
query16	0.36	0.36	0.38
query17	1.02	1.01	1.00
query18	0.18	0.28	0.24
query19	1.89	1.86	1.85
query20	0.01	0.01	0.01
query21	15.48	0.56	0.54
query22	2.01	1.83	1.64
query23	17.34	0.97	0.86
query24	6.95	0.98	1.10
query25	0.37	0.16	0.04
query26	0.72	0.14	0.15
query27	0.04	0.04	0.03
query28	6.15	0.75	0.80
query29	12.62	2.36	2.19
query30	0.61	0.54	0.54
query31	2.81	0.39	0.37
query32	3.37	0.49	0.50
query33	3.08	3.09	3.11
query34	15.26	4.79	4.78
query35	4.88	4.83	4.86
query36	1.04	1.02	1.01
query37	0.06	0.05	0.04
query38	0.04	0.03	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.14
query41	0.08	0.01	0.01
query42	0.02	0.01	0.01
query43	0.04	0.02	0.02
Total cold run time: 104.09 s
Total hot run time: 30.66 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3fee58c448c05be7b8b3e4921b554c42c4e5d4d2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@xiaokang
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50075 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c425e3e767cbaa238c2d0daa9ef6489e93eae4a9, data reload: false

------ Round 1 ----------------------------------
q1	17664	4428	4372	4372
q2	2076	169	148	148
q3	10257	1921	1915	1915
q4	10332	1271	1346	1271
q5	8408	3921	3949	3921
q6	256	125	126	125
q7	2029	1634	1585	1585
q8	9507	2744	2737	2737
q9	13986	10373	10168	10168
q10	8683	3542	3564	3542
q11	407	254	247	247
q12	471	312	315	312
q13	18369	3953	4030	3953
q14	359	327	329	327
q15	497	464	456	456
q16	657	581	578	578
q17	1145	969	919	919
q18	8273	6961	6959	6959
q19	1778	1702	1680	1680
q20	576	315	307	307
q21	4530	4136	4120	4120
q22	535	447	433	433
Total cold run time: 120795 ms
Total hot run time: 50075 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4346	4285	4285
q2	323	234	221	221
q3	4219	4158	4129	4129
q4	2771	2746	2748	2746
q5	7251	7153	7166	7153
q6	237	120	121	120
q7	3334	2888	2856	2856
q8	4460	4532	4499	4499
q9	16870	16760	16782	16760
q10	4237	4269	4275	4269
q11	756	675	697	675
q12	1038	849	866	849
q13	7047	3751	3803	3751
q14	455	422	423	422
q15	506	461	466	461
q16	740	700	698	698
q17	3813	3918	3921	3918
q18	8903	8742	8912	8742
q19	1754	1740	1728	1728
q20	2381	2154	2077	2077
q21	8666	8549	8520	8520
q22	1054	967	1002	967
Total cold run time: 85163 ms
Total hot run time: 79846 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c425e3e767cbaa238c2d0daa9ef6489e93eae4a9, data reload: false

query1	935	412	378	378
query2	6526	2839	2538	2538
query3	6940	201	201	201
query4	19788	17982	17981	17981
query5	19740	6449	6496	6449
query6	301	214	241	214
query7	4151	306	313	306
query8	419	428	414	414
query9	3139	2658	2601	2601
query10	415	306	300	300
query11	11288	10687	10641	10641
query12	132	83	74	74
query13	5596	707	707	707
query14	17768	13257	13776	13257
query15	364	244	232	232
query16	6480	292	273	273
query17	1728	1469	887	887
query18	2312	427	419	419
query19	218	156	151	151
query20	81	81	85	81
query21	188	99	91	91
query22	5254	5041	5040	5040
query23	32655	32105	31976	31976
query24	6964	6567	6511	6511
query25	530	423	435	423
query26	528	168	162	162
query27	1852	299	290	290
query28	6224	2392	2337	2337
query29	2839	2788	2846	2788
query30	238	167	169	167
query31	893	759	777	759
query32	75	64	59	59
query33	398	264	259	259
query34	844	476	498	476
query35	1082	903	927	903
query36	1215	1254	1120	1120
query37	96	60	63	60
query38	3109	2907	2929	2907
query39	1393	1326	1325	1325
query40	205	99	98	98
query41	47	44	44	44
query42	84	85	80	80
query43	742	700	723	700
query44	1169	731	732	731
query45	252	238	233	233
query46	1232	969	998	969
query47	1888	1762	1717	1717
query48	1021	724	706	706
query49	620	378	383	378
query50	876	650	636	636
query51	4864	4687	4682	4682
query52	104	85	87	85
query53	452	320	332	320
query54	2639	2477	2472	2472
query55	87	79	88	79
query56	251	213	220	213
query57	1233	1068	1157	1068
query58	221	210	208	208
query59	4360	4173	3954	3954
query60	213	187	225	187
query61	99	92	95	92
query62	791	476	590	476
query63	482	340	342	340
query64	2606	1497	1442	1442
query65	3635	3600	3576	3576
query66	847	388	389	388
query67	15667	15430	15525	15430
query68	10093	675	653	653
query69	582	361	352	352
query70	1821	1396	1427	1396
query71	407	303	311	303
query72	6549	3454	3484	3454
query73	729	318	322	318
query74	6223	5826	5830	5826
query75	5413	3617	3637	3617
query76	6485	1142	1193	1142
query77	1071	263	254	254
query78	12825	11645	11699	11645
query79	9105	646	665	646
query80	1117	406	411	406
query81	494	233	234	233
query82	1576	96	107	96
query83	173	138	133	133
query84	257	72	71	71
query85	859	331	336	331
query86	353	299	289	289
query87	3257	3037	3022	3022
query88	4739	2334	2332	2332
query89	470	311	296	296
query90	1942	217	210	210
query91	175	139	138	138
query92	63	57	51	51
query93	7001	583	580	580
query94	706	211	211	211
query95	1122	1070	1078	1070
query96	640	330	332	330
query97	6382	6381	6233	6233
query98	189	185	178	178
query99	2966	992	899	899
Total cold run time: 317169 ms
Total hot run time: 203776 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c425e3e767cbaa238c2d0daa9ef6489e93eae4a9, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.06	0.04
query4	1.79	0.06	0.06
query5	0.54	0.51	0.53
query6	1.23	0.67	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.49	0.48
query10	0.54	0.53	0.54
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.79	0.78
query15	0.77	0.76	0.77
query16	0.36	0.37	0.37
query17	1.01	1.00	0.98
query18	0.24	0.22	0.26
query19	1.90	1.78	1.84
query20	0.02	0.01	0.01
query21	15.47	0.55	0.57
query22	2.04	2.78	1.53
query23	17.55	0.98	0.98
query24	5.56	5.38	0.68
query25	0.40	0.13	0.05
query26	0.99	0.18	0.14
query27	0.04	0.04	0.04
query28	3.61	0.77	0.74
query29	12.65	2.33	2.16
query30	0.60	0.49	0.54
query31	2.82	0.39	0.37
query32	3.40	0.50	0.49
query33	3.07	3.07	3.08
query34	15.26	4.76	4.80
query35	4.84	4.82	4.80
query36	1.05	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 100.67 s
Total hot run time: 30.11 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.73% (8100/21466)
Line Coverage: 29.38% (66329/225799)
Region Coverage: 28.88% (34221/118477)
Branch Coverage: 24.78% (17586/70982)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c425e3e767cbaa238c2d0daa9ef6489e93eae4a9_c425e3e767cbaa238c2d0daa9ef6489e93eae4a9/report/index.html

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c425e3e767cbaa238c2d0daa9ef6489e93eae4a9 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@xiaokang xiaokang merged commit 6ddca2a into apache:branch-2.0 Aug 1, 2024
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/nereids area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@keanji-x @doris-robot @xiaokang