Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](OrcWriter) fix be core when upgrading BE without upgrading FE #40282

Merged
merged 7 commits into from
Sep 4, 2024
Merged

[fix](OrcWriter) fix be core when upgrading BE without upgrading FE #40282

merged 7 commits into from
Sep 4, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Sep 2, 2024 •
edited
Loading

Since we have changed the type mapping from Doris to Orc type, using the Outfile to export Date/Datetime types will cause BE core dump when only upgrading BE without upgrading FE.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

morningman
morningman previously approved these changes Sep 2, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

PR approved by anyone and no changes requested.

@BePPPower BePPPower marked this pull request as ready for review September 3, 2024 03:57
@BePPPower BePPPower changed the title [fix](OrcWriter) fix be core when upgrade BE [fix](OrcWriter) fix be core when upgrade BE without upgraded FE Sep 3, 2024
@BePPPower
Copy link
Contributor Author

run buildall

@BePPPower BePPPower changed the title [fix](OrcWriter) fix be core when upgrade BE without upgraded FE [fix](OrcWriter) fix be core when upgrading BE without upgrading FE Sep 3, 2024
@BePPPower BePPPower mentioned this pull request Sep 3, 2024
Merged
@doris-robot
Copy link

TPC-H: Total hot run time: 38334 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7da2dd262ac1c33969334ffefc46c670f36ad1a7, data reload: false

------ Round 1 ----------------------------------
q1	18071	5333	4302	4302
q2	2044	187	185	185
q3	11448	974	1106	974
q4	10148	772	675	675
q5	7787	2835	2804	2804
q6	231	140	136	136
q7	971	617	590	590
q8	9347	2106	2114	2106
q9	7269	6585	6553	6553
q10	7016	2263	2346	2263
q11	466	244	243	243
q12	401	220	221	220
q13	18730	3073	3100	3073
q14	290	240	243	240
q15	527	501	491	491
q16	570	521	512	512
q17	1001	680	728	680
q18	7438	6875	7014	6875
q19	1393	1025	969	969
q20	683	333	334	333
q21	4031	3233	3103	3103
q22	1123	1033	1007	1007
Total cold run time: 110985 ms
Total hot run time: 38334 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4326	4306	4261	4261
q2	396	279	266	266
q3	2923	2731	2676	2676
q4	1937	1688	1773	1688
q5	5686	5703	5809	5703
q6	225	131	131	131
q7	2254	1852	1851	1851
q8	3325	3460	3511	3460
q9	8878	8812	8845	8812
q10	3666	3400	3330	3330
q11	593	504	510	504
q12	824	700	670	670
q13	13824	3120	3259	3120
q14	326	301	288	288
q15	534	492	487	487
q16	643	594	582	582
q17	1875	1555	1528	1528
q18	8176	7850	7821	7821
q19	1754	1753	1567	1567
q20	2122	1945	1913	1913
q21	5882	5574	5558	5558
q22	1139	1068	1030	1030
Total cold run time: 71308 ms
Total hot run time: 57246 ms

@morningman
Copy link
Contributor

run buildall

@BePPPower
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39133 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef2f8bc0c89257fae048248898dd61c2f6bd79a7, data reload: false

------ Round 1 ----------------------------------
q1	17663	4650	4506	4506
q2	2027	202	215	202
q3	10416	1127	1166	1127
q4	10161	802	759	759
q5	7449	1139	1101	1101
q6	204	150	152	150
q7	923	539	518	518
q8	9355	1390	1426	1390
q9	8692	6909	6915	6909
q10	9785	4770	4850	4770
q11	460	271	281	271
q12	516	253	252	252
q13	18955	3119	3100	3100
q14	484	455	430	430
q15	586	555	542	542
q16	678	583	544	544
q17	995	691	692	691
q18	7540	7066	7026	7026
q19	1400	1009	1068	1009
q20	717	371	379	371
q21	3648	3357	3157	3157
q22	383	314	308	308
Total cold run time: 113037 ms
Total hot run time: 39133 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4322	4302	4302
q2	532	444	433	433
q3	2879	2682	2666	2666
q4	1962	1692	1776	1692
q5	5668	5637	5684	5637
q6	230	134	134	134
q7	8698	8168	8203	8168
q8	3675	3801	3948	3801
q9	17754	17835	18214	17835
q10	5425	5168	5157	5157
q11	1325	1125	1206	1125
q12	864	639	660	639
q13	9988	3292	3303	3292
q14	462	446	432	432
q15	578	528	540	528
q16	735	685	659	659
q17	3191	2783	2765	2765
q18	8303	7918	7691	7691
q19	1757	1642	1590	1590
q20	2133	1850	1870	1850
q21	12179	12012	11743	11743
q22	616	527	500	500
Total cold run time: 93371 ms
Total hot run time: 82639 ms

@BePPPower
Copy link
Contributor Author

run buildall

1 similar comment
@BePPPower
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38127 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85e54892c23060f8846607276d7a5c0540c493a3, data reload: false

------ Round 1 ----------------------------------
q1	17629	4496	4270	4270
q2	2016	178	179	178
q3	11868	973	1147	973
q4	10142	730	782	730
q5	7759	2841	2853	2841
q6	226	140	139	139
q7	960	622	593	593
q8	9316	2078	2092	2078
q9	7311	6497	6541	6497
q10	6990	2243	2191	2191
q11	462	241	242	241
q12	396	222	217	217
q13	17762	3035	3044	3035
q14	272	239	228	228
q15	532	490	495	490
q16	587	505	499	499
q17	973	694	676	676
q18	7221	6940	6877	6877
q19	1385	1025	1071	1025
q20	675	335	328	328
q21	3862	2990	3124	2990
q22	1140	1031	1036	1031
Total cold run time: 109484 ms
Total hot run time: 38127 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4279	4235	4235
q2	384	275	267	267
q3	2871	2681	2646	2646
q4	1936	1614	1650	1614
q5	5575	5719	5692	5692
q6	221	135	145	135
q7	2232	1800	1830	1800
q8	3276	3398	3393	3393
q9	8842	8790	8795	8790
q10	3565	3310	3366	3310
q11	592	508	529	508
q12	834	642	647	642
q13	11390	3163	3283	3163
q14	331	290	287	287
q15	540	490	477	477
q16	650	585	577	577
q17	1812	1521	1519	1519
q18	8083	7753	7922	7753
q19	1709	1596	1527	1527
q20	2198	1921	1969	1921
q21	5711	5539	5485	5485
q22	1147	1075	1069	1069
Total cold run time: 68256 ms
Total hot run time: 56810 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193547 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85e54892c23060f8846607276d7a5c0540c493a3, data reload: false

query1	1248	898	869	869
query2	6161	1865	1874	1865
query3	10693	4086	3980	3980
query4	59882	25958	23322	23322
query5	5434	500	485	485
query6	412	159	168	159
query7	5772	291	285	285
query8	296	214	207	207
query9	9054	2468	2453	2453
query10	493	274	261	261
query11	18359	16409	15375	15375
query12	150	103	104	103
query13	1567	397	391	391
query14	10936	7580	7427	7427
query15	220	185	181	181
query16	7568	501	473	473
query17	1103	554	557	554
query18	2058	296	290	290
query19	285	149	148	148
query20	118	115	112	112
query21	203	108	108	108
query22	5002	4776	4849	4776
query23	34211	33244	33466	33244
query24	5925	2832	2858	2832
query25	525	372	375	372
query26	680	152	151	151
query27	1784	268	277	268
query28	3849	2025	1995	1995
query29	691	408	420	408
query30	238	153	159	153
query31	912	766	764	764
query32	85	51	62	51
query33	486	285	269	269
query34	845	486	478	478
query35	828	722	731	722
query36	1092	923	915	915
query37	148	87	94	87
query38	3928	3898	3978	3898
query39	1452	1396	1393	1393
query40	207	116	114	114
query41	49	43	41	41
query42	114	94	90	90
query43	508	456	474	456
query44	1085	724	722	722
query45	198	168	163	163
query46	1098	731	756	731
query47	1943	1847	1829	1829
query48	361	281	292	281
query49	809	420	422	420
query50	802	416	406	406
query51	6954	6821	6905	6821
query52	96	85	83	83
query53	247	178	177	177
query54	553	448	448	448
query55	72	72	74	72
query56	281	261	268	261
query57	1191	1096	1082	1082
query58	222	219	225	219
query59	2916	2811	2630	2630
query60	288	271	274	271
query61	110	98	101	98
query62	738	640	644	640
query63	229	179	189	179
query64	2813	688	675	675
query65	3198	3146	3136	3136
query66	689	356	338	338
query67	15444	15319	15245	15245
query68	3005	576	561	561
query69	398	289	279	279
query70	1141	1064	1121	1064
query71	347	277	277	277
query72	5389	3996	3993	3993
query73	744	331	330	330
query74	9215	8876	8825	8825
query75	3332	2672	2663	2663
query76	1422	1066	999	999
query77	528	315	319	315
query78	9762	9710	9562	9562
query79	1065	522	528	522
query80	709	538	501	501
query81	545	234	234	234
query82	747	136	138	136
query83	168	148	148	148
query84	269	80	76	76
query85	711	284	314	284
query86	312	303	305	303
query87	4324	4212	4220	4212
query88	2917	2323	2319	2319
query89	384	289	285	285
query90	1713	188	189	188
query91	118	98	102	98
query92	58	49	51	49
query93	1071	529	526	526
query94	555	283	291	283
query95	344	257	252	252
query96	592	265	270	265
query97	3287	3069	3072	3069
query98	217	200	209	200
query99	1702	1277	1284	1277
Total cold run time: 304999 ms
Total hot run time: 193547 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85e54892c23060f8846607276d7a5c0540c493a3, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.07
query5	0.51	0.52	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.58
query14	2.05	2.03	2.06
query15	0.85	0.83	0.82
query16	0.38	0.38	0.38
query17	1.00	1.04	1.04
query18	0.22	0.20	0.20
query19	1.83	1.85	1.76
query20	0.01	0.01	0.01
query21	15.43	0.68	0.67
query22	4.34	7.81	2.10
query23	18.24	1.35	1.25
query24	2.15	0.21	0.23
query25	0.15	0.08	0.08
query26	0.27	0.19	0.17
query27	0.08	0.07	0.07
query28	13.19	1.02	1.00
query29	12.61	3.32	3.26
query30	0.25	0.06	0.05
query31	2.89	0.40	0.39
query32	3.25	0.49	0.48
query33	2.95	3.05	3.01
query34	17.13	4.44	4.41
query35	4.46	4.59	4.49
query36	0.67	0.48	0.51
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.06	0.04	0.04
Total cold run time: 110.93 s
Total hot run time: 32.27 s

@BePPPower
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@morningman
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

yiguolei pushed a commit that referenced this pull request Sep 4, 2024
@BePPPower
[fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#…
b4beec8 
…40303)

bp: #40282
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38403 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dcc97611f6b5c84c16536723b6c22003aa96f4ad, data reload: false

------ Round 1 ----------------------------------
q1	17850	4556	4408	4408
q2	2369	184	180	180
q3	10473	1214	1104	1104
q4	10132	686	831	686
q5	7729	2900	2772	2772
q6	230	140	140	140
q7	964	637	609	609
q8	9326	2084	2087	2084
q9	7098	6567	6549	6549
q10	6998	2189	2359	2189
q11	465	248	252	248
q12	406	229	222	222
q13	18901	3150	3134	3134
q14	278	249	241	241
q15	548	508	482	482
q16	591	514	511	511
q17	986	754	712	712
q18	7558	6889	6984	6889
q19	1401	1048	1069	1048
q20	683	346	334	334
q21	4010	3016	2862	2862
q22	1124	999	1008	999
Total cold run time: 110120 ms
Total hot run time: 38403 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4333	4328	4328
q2	371	286	278	278
q3	2935	2664	2710	2664
q4	1958	1680	1670	1670
q5	5435	5443	5435	5435
q6	221	134	135	134
q7	2133	1729	1775	1729
q8	3256	3372	3362	3362
q9	8570	8456	8480	8456
q10	3465	3215	3192	3192
q11	598	503	498	498
q12	803	600	603	600
q13	14068	3097	3130	3097
q14	299	277	275	275
q15	527	487	479	479
q16	622	589	582	582
q17	1800	1505	1503	1503
q18	7848	7510	7479	7479
q19	1666	1652	1430	1430
q20	2088	1837	1838	1837
q21	5392	5246	5163	5163
q22	1134	1091	1023	1023
Total cold run time: 69584 ms
Total hot run time: 55214 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187175 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dcc97611f6b5c84c16536723b6c22003aa96f4ad, data reload: false

query1	911	382	373	373
query2	6466	2003	1971	1971
query3	6665	214	219	214
query4	34322	23209	23560	23209
query5	4178	504	516	504
query6	266	173	174	173
query7	4596	292	289	289
query8	251	197	214	197
query9	8871	2514	2508	2508
query10	430	291	292	291
query11	17860	15035	15203	15035
query12	154	100	97	97
query13	1629	387	380	380
query14	10025	7380	7408	7380
query15	272	178	182	178
query16	8088	439	507	439
query17	1586	593	573	573
query18	2130	308	286	286
query19	330	146	146	146
query20	118	110	104	104
query21	210	107	103	103
query22	4352	4025	4086	4025
query23	34119	33443	33303	33303
query24	11224	2882	2933	2882
query25	635	410	393	393
query26	1153	161	162	161
query27	2785	287	283	283
query28	7654	2064	2052	2052
query29	825	417	408	408
query30	310	165	159	159
query31	986	772	795	772
query32	105	57	63	57
query33	809	312	300	300
query34	929	476	492	476
query35	849	730	738	730
query36	1099	897	928	897
query37	165	96	91	91
query38	3994	3911	3808	3808
query39	1438	1411	1404	1404
query40	198	122	117	117
query41	50	49	45	45
query42	124	102	99	99
query43	514	483	481	481
query44	1298	758	740	740
query45	199	167	167	167
query46	1111	738	737	737
query47	1877	1765	1782	1765
query48	390	309	301	301
query49	1087	441	453	441
query50	821	419	429	419
query51	6996	6884	6923	6884
query52	102	125	85	85
query53	264	186	184	184
query54	944	472	465	465
query55	78	74	79	74
query56	298	253	258	253
query57	1214	1072	1051	1051
query58	258	232	245	232
query59	2956	2702	2794	2702
query60	317	285	282	282
query61	109	101	103	101
query62	835	655	657	655
query63	226	196	186	186
query64	5083	689	666	666
query65	3273	3147	3203	3147
query66	1424	349	352	349
query67	15701	15316	15385	15316
query68	3121	595	589	589
query69	390	288	290	288
query70	1096	1126	1098	1098
query71	356	277	279	277
query72	6249	4042	2581	2581
query73	780	331	331	331
query74	9107	8763	8828	8763
query75	3417	2713	2688	2688
query76	1882	961	966	961
query77	492	343	324	324
query78	11174	10547	9251	9251
query79	1118	550	545	545
query80	766	523	530	523
query81	565	237	245	237
query82	648	156	149	149
query83	238	157	158	157
query84	230	83	85	83
query85	718	301	327	301
query86	306	307	293	293
query87	4429	4276	4287	4276
query88	3134	2347	2348	2347
query89	390	294	288	288
query90	2141	196	194	194
query91	132	109	105	105
query92	65	48	54	48
query93	1072	557	549	549
query94	880	303	283	283
query95	359	256	262	256
query96	597	269	265	265
query97	3215	3072	3104	3072
query98	218	207	194	194
query99	1568	1264	1298	1264
Total cold run time: 291389 ms
Total hot run time: 187175 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dcc97611f6b5c84c16536723b6c22003aa96f4ad, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.51	0.50	0.51
query6	1.14	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.52	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.03	2.03	2.07
query15	0.92	0.83	0.81
query16	0.37	0.38	0.36
query17	1.07	1.06	1.05
query18	0.21	0.20	0.21
query19	1.95	1.87	1.89
query20	0.02	0.01	0.01
query21	15.41	0.67	0.67
query22	4.02	7.84	1.78
query23	18.24	1.42	1.24
query24	2.08	0.22	0.22
query25	0.13	0.07	0.08
query26	0.28	0.18	0.17
query27	0.08	0.07	0.08
query28	13.28	1.02	0.99
query29	12.64	3.31	3.25
query30	0.24	0.06	0.06
query31	2.86	0.40	0.40
query32	3.22	0.50	0.48
query33	2.97	2.97	3.03
query34	17.28	4.42	4.40
query35	4.42	4.40	4.43
query36	0.67	0.48	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.83 s
Total hot run time: 31.92 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

@hello-stephen hello-stephen merged commit edb6eec into apache:master Sep 4, 2024
24 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@BePPPower @dataroaring
[fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#…
54a8dec 
…40282)

Since we have changed the type mapping from Doris to Orc type, using the
`Outfile` to export Date/Datetime types will cause BE core dump when
only upgrading BE without upgrading FE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman left review comments

@hello-stephen hello-stephen hello-stephen approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@BePPPower @doris-robot @morningman @hello-stephen @CalvinKirs @yiguolei