Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[minor](dependencies)binding a third-party Hadoop component to version 1.2.0 #35093

Merged
merged 7 commits into from
Jun 11, 2024
Merged

[minor](dependencies)binding a third-party Hadoop component to version 1.2.0 #35093

merged 7 commits into from
Jun 11, 2024

Conversation

CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented May 21, 2024 •
edited
Loading

Proposed changes

binding a third-party Hadoop component to version 1.2.0

hadoop-shaded-protobuf and hadoop-shaded-guava are older components, and the components such as guava they depend on have some known CVE issues, so they were upgraded to 1.2.0

remove unless hadoop yarn and htrace jars

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39932 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c53fcdebce33619e0893aeb8ecccaf5567b5596, data reload: false

------ Round 1 ----------------------------------
q1	18010	4469	4366	4366
q2	2462	189	201	189
q3	11623	1207	1244	1207
q4	10695	777	820	777
q5	7589	2717	2704	2704
q6	228	134	137	134
q7	967	614	599	599
q8	9270	2042	2046	2042
q9	8884	6507	6489	6489
q10	8841	3651	3702	3651
q11	448	242	226	226
q12	472	222	218	218
q13	17757	2966	2964	2964
q14	253	225	219	219
q15	511	476	475	475
q16	532	376	370	370
q17	954	650	595	595
q18	8074	7418	7408	7408
q19	5004	1565	1481	1481
q20	647	309	315	309
q21	4865	3888	3236	3236
q22	339	273	283	273
Total cold run time: 118425 ms
Total hot run time: 39932 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4316	4230	4230
q2	377	269	272	269
q3	2956	2799	2677	2677
q4	1833	1616	1558	1558
q5	5251	5234	5232	5232
q6	221	127	128	127
q7	2107	1763	1721	1721
q8	3161	3606	3303	3303
q9	8359	8359	8282	8282
q10	3890	3660	3726	3660
q11	595	496	487	487
q12	751	592	614	592
q13	16398	2956	2987	2956
q14	290	256	269	256
q15	519	469	474	469
q16	471	412	416	412
q17	1770	1491	1472	1472
q18	7710	7380	7372	7372
q19	1650	1533	1483	1483
q20	1981	1774	1770	1770
q21	4821	4787	4776	4776
q22	589	495	492	492
Total cold run time: 70083 ms
Total hot run time: 53596 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171831 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c53fcdebce33619e0893aeb8ecccaf5567b5596, data reload: false

query1	928	380	376	376
query2	6458	2263	2310	2263
query3	6653	204	207	204
query4	19162	17534	17377	17377
query5	4165	421	438	421
query6	255	153	159	153
query7	4589	298	282	282
query8	233	186	182	182
query9	8603	2367	2327	2327
query10	442	274	255	255
query11	10734	10198	10212	10198
query12	132	93	89	89
query13	1636	351	362	351
query14	10129	6809	6625	6625
query15	216	169	173	169
query16	7826	269	251	251
query17	1616	534	529	529
query18	1955	274	264	264
query19	216	151	152	151
query20	90	85	83	83
query21	209	129	127	127
query22	4260	4108	4008	4008
query23	33508	32979	32904	32904
query24	12078	2875	2874	2874
query25	635	345	361	345
query26	1628	149	152	149
query27	2999	318	321	318
query28	7553	2042	2043	2042
query29	1037	601	601	601
query30	307	172	175	172
query31	980	762	753	753
query32	97	51	53	51
query33	756	264	253	253
query34	1020	496	472	472
query35	728	598	581	581
query36	1070	907	878	878
query37	141	70	71	70
query38	2879	2794	2776	2776
query39	861	789	797	789
query40	270	121	121	121
query41	49	45	44	44
query42	100	98	93	93
query43	579	559	559	559
query44	1240	724	736	724
query45	178	162	162	162
query46	1073	718	725	718
query47	1848	1770	1840	1770
query48	360	287	280	280
query49	1152	375	381	375
query50	764	369	370	369
query51	6849	6817	6716	6716
query52	100	90	89	89
query53	346	284	291	284
query54	876	423	454	423
query55	72	70	69	69
query56	255	239	239	239
query57	1113	1053	1068	1053
query58	249	203	208	203
query59	3352	3169	3128	3128
query60	264	251	247	247
query61	90	85	83	83
query62	666	453	436	436
query63	312	288	283	283
query64	9813	2281	1727	1727
query65	3174	3092	3089	3089
query66	1379	328	321	321
query67	15210	15225	15000	15000
query68	4543	523	528	523
query69	430	261	265	261
query70	1171	1142	1114	1114
query71	402	269	266	266
query72	7348	5439	5453	5439
query73	721	317	313	313
query74	6211	5618	5621	5618
query75	3324	2599	2588	2588
query76	2423	1029	930	930
query77	417	260	260	260
query78	10310	9671	9740	9671
query79	2055	516	505	505
query80	878	427	428	427
query81	519	240	242	240
query82	649	94	92	92
query83	242	168	164	164
query84	233	85	84	84
query85	1871	256	257	256
query86	528	313	306	306
query87	3322	3146	3150	3146
query88	4073	2308	2289	2289
query89	483	398	370	370
query90	1959	186	185	185
query91	131	109	103	103
query92	60	49	52	49
query93	2357	509	501	501
query94	1255	194	253	194
query95	396	303	300	300
query96	595	259	264	259
query97	3160	2982	3042	2982
query98	244	215	216	215
query99	1101	863	864	863
Total cold run time: 273928 ms
Total hot run time: 171831 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c53fcdebce33619e0893aeb8ecccaf5567b5596, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.07	0.06
query4	1.66	0.09	0.10
query5	0.51	0.49	0.52
query6	1.12	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.53	0.49	0.49
query10	0.55	0.53	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.61
query14	0.78	0.77	0.79
query15	0.82	0.80	0.82
query16	0.36	0.37	0.36
query17	0.94	1.01	0.96
query18	0.23	0.26	0.24
query19	1.76	1.71	1.77
query20	0.01	0.01	0.02
query21	15.44	0.70	0.67
query22	4.88	6.84	1.95
query23	18.31	1.40	1.22
query24	1.96	0.23	0.21
query25	0.15	0.09	0.08
query26	0.26	0.17	0.18
query27	0.08	0.07	0.08
query28	13.25	1.02	1.00
query29	12.61	3.27	3.30
query30	0.24	0.07	0.05
query31	2.87	0.39	0.38
query32	3.28	0.47	0.47
query33	2.84	2.86	2.93
query34	17.04	4.52	4.43
query35	4.49	4.45	4.46
query36	0.64	0.46	0.45
query37	0.17	0.15	0.16
query38	0.14	0.14	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.04	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.63 s
Total hot run time: 30.48 s

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40872 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b73f2549e45cb9928b4a39b084f4939152564e79, data reload: false

------ Round 1 ----------------------------------
q1	17595	4258	4210	4210
q2	2013	188	188	188
q3	10455	1204	1235	1204
q4	10193	883	790	790
q5	7476	2714	2660	2660
q6	226	129	131	129
q7	947	612	613	612
q8	9236	2100	2070	2070
q9	8999	6661	6657	6657
q10	9207	3955	3903	3903
q11	448	252	239	239
q12	418	224	225	224
q13	17401	3227	3239	3227
q14	253	212	215	212
q15	496	466	484	466
q16	472	384	395	384
q17	953	655	766	655
q18	8438	7818	7689	7689
q19	5431	1556	1541	1541
q20	621	317	319	317
q21	5116	3219	3268	3219
q22	341	276	279	276
Total cold run time: 116735 ms
Total hot run time: 40872 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4445	4369	4351	4351
q2	387	279	277	277
q3	3137	2983	2874	2874
q4	1972	1653	1611	1611
q5	5333	5490	5470	5470
q6	221	124	124	124
q7	2179	1806	1814	1806
q8	3208	3404	3368	3368
q9	8516	8537	8675	8537
q10	4017	3771	3684	3684
q11	608	487	508	487
q12	789	630	625	625
q13	17078	3128	3172	3128
q14	292	268	289	268
q15	526	479	470	470
q16	511	447	425	425
q17	1779	1529	1504	1504
q18	7783	7602	7463	7463
q19	1647	1563	1562	1562
q20	2013	1754	1789	1754
q21	10699	4683	4718	4683
q22	547	481	473	473
Total cold run time: 77687 ms
Total hot run time: 54944 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168090 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b73f2549e45cb9928b4a39b084f4939152564e79, data reload: false

query1	910	373	372	372
query2	6446	2415	2331	2331
query3	6642	207	215	207
query4	20082	17246	17335	17246
query5	4121	410	416	410
query6	246	155	150	150
query7	4581	296	284	284
query8	250	179	182	179
query9	8658	2383	2358	2358
query10	446	285	265	265
query11	10704	10056	10004	10004
query12	135	97	90	90
query13	1644	387	351	351
query14	9439	6509	7509	6509
query15	210	166	175	166
query16	7747	257	268	257
query17	1624	521	516	516
query18	1938	265	269	265
query19	208	154	161	154
query20	91	88	81	81
query21	195	133	122	122
query22	4422	3964	3903	3903
query23	33638	33075	32653	32653
query24	6887	2896	2798	2798
query25	569	351	349	349
query26	710	160	155	155
query27	2031	318	331	318
query28	3933	2035	2029	2029
query29	849	604	608	604
query30	263	170	175	170
query31	966	779	756	756
query32	97	50	52	50
query33	484	267	255	255
query34	849	471	460	460
query35	716	589	595	589
query36	1055	926	922	922
query37	106	67	69	67
query38	2898	2785	2737	2737
query39	858	781	792	781
query40	194	124	120	120
query41	48	42	42	42
query42	103	95	100	95
query43	584	572	541	541
query44	1080	722	751	722
query45	183	164	167	164
query46	1059	703	783	703
query47	1858	1794	1815	1794
query48	361	301	288	288
query49	818	372	382	372
query50	754	379	376	376
query51	6870	6697	6805	6697
query52	102	88	90	88
query53	347	275	291	275
query54	534	433	416	416
query55	74	74	72	72
query56	249	236	263	236
query57	1142	1053	1066	1053
query58	225	211	207	207
query59	3397	3207	3247	3207
query60	309	248	244	244
query61	85	89	84	84
query62	588	458	443	443
query63	303	279	278	278
query64	8390	2211	1754	1754
query65	3203	3079	3095	3079
query66	785	319	326	319
query67	15034	14646	14760	14646
query68	4463	529	530	529
query69	436	259	274	259
query70	1177	1066	1163	1066
query71	374	268	263	263
query72	7700	5491	2735	2735
query73	704	321	326	321
query74	6000	5695	5593	5593
query75	3303	2626	2609	2609
query76	2309	991	968	968
query77	388	272	267	267
query78	10304	9934	9750	9750
query79	2432	506	496	496
query80	1018	432	425	425
query81	533	244	242	242
query82	1278	91	95	91
query83	241	166	169	166
query84	247	99	87	87
query85	1287	266	256	256
query86	482	304	283	283
query87	3347	3104	3145	3104
query88	4080	2356	2340	2340
query89	454	383	369	369
query90	2067	194	182	182
query91	119	94	146	94
query92	63	46	48	46
query93	2677	515	502	502
query94	1308	180	185	180
query95	399	303	307	303
query96	589	264	267	264
query97	3187	3000	3045	3000
query98	244	223	217	217
query99	1111	871	855	855
Total cold run time: 260648 ms
Total hot run time: 168090 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b73f2549e45cb9928b4a39b084f4939152564e79, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.48
query6	1.11	0.71	0.74
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.54	0.49	0.49
query10	0.54	0.54	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.76	0.78
query15	0.83	0.81	0.81
query16	0.36	0.36	0.37
query17	0.93	0.99	1.02
query18	0.22	0.26	0.22
query19	1.73	1.68	1.66
query20	0.01	0.02	0.01
query21	15.50	0.69	0.67
query22	4.46	7.37	1.66
query23	18.25	1.35	1.31
query24	1.62	0.32	0.19
query25	0.14	0.09	0.08
query26	0.26	0.15	0.17
query27	0.07	0.08	0.07
query28	13.36	1.01	0.99
query29	12.68	3.27	3.24
query30	0.24	0.05	0.08
query31	2.84	0.38	0.39
query32	3.30	0.46	0.46
query33	2.85	2.98	2.83
query34	17.16	4.41	4.42
query35	4.49	4.53	4.49
query36	0.69	0.49	0.48
query37	0.18	0.15	0.14
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.04	0.04	0.03
Total cold run time: 109.19 s
Total hot run time: 30.14 s

@CalvinKirs
Copy link
Member Author

run p0 (timeout)

@CalvinKirs
Copy link
Member Author

run p0

@CalvinKirs
Copy link
Member Author

run P0

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40104 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

------ Round 1 ----------------------------------
q1	18662	4531	4432	4432
q2	2536	197	193	193
q3	11065	1154	1208	1154
q4	10644	786	763	763
q5	7613	2682	2688	2682
q6	224	137	144	137
q7	989	607	616	607
q8	9468	2072	2067	2067
q9	8766	6490	6468	6468
q10	8938	3689	3690	3689
q11	448	245	236	236
q12	449	224	216	216
q13	17766	2984	2969	2969
q14	259	218	224	218
q15	511	467	468	467
q16	523	384	380	380
q17	965	649	712	649
q18	8121	7499	7404	7404
q19	4509	1559	1483	1483
q20	648	315	328	315
q21	5038	3294	3752	3294
q22	346	281	293	281
Total cold run time: 118488 ms
Total hot run time: 40104 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4193	4229	4193
q2	368	257	262	257
q3	3009	2801	2733	2733
q4	1871	1587	1610	1587
q5	5260	5266	5274	5266
q6	209	124	130	124
q7	2129	1759	1801	1759
q8	3227	3322	3311	3311
q9	8356	8364	8305	8305
q10	3925	3719	3650	3650
q11	622	496	487	487
q12	766	581	598	581
q13	17223	3039	3035	3035
q14	299	276	266	266
q15	514	477	484	477
q16	476	415	418	415
q17	1775	1491	1480	1480
q18	7786	7483	7563	7483
q19	1691	1542	1579	1542
q20	2018	1788	1817	1788
q21	4790	4699	4834	4699
q22	583	493	478	478
Total cold run time: 71251 ms
Total hot run time: 53916 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173369 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

query1	910	382	369	369
query2	6461	2502	2395	2395
query3	6667	209	210	209
query4	19786	17315	17267	17267
query5	4165	419	426	419
query6	250	162	159	159
query7	4589	304	297	297
query8	236	188	177	177
query9	8521	2405	2387	2387
query10	443	279	254	254
query11	10742	9904	9901	9901
query12	138	90	85	85
query13	1633	368	353	353
query14	10066	7949	7636	7636
query15	215	178	171	171
query16	7558	265	268	265
query17	1850	521	527	521
query18	1228	276	262	262
query19	216	159	151	151
query20	93	88	83	83
query21	193	128	127	127
query22	4180	3944	3961	3944
query23	33720	33124	32959	32959
query24	12043	2859	2858	2858
query25	693	372	393	372
query26	1656	158	161	158
query27	2691	326	324	324
query28	7170	2045	2077	2045
query29	1046	643	612	612
query30	329	173	182	173
query31	971	767	748	748
query32	92	55	57	55
query33	837	266	261	261
query34	1012	471	471	471
query35	745	590	587	587
query36	1092	913	922	913
query37	238	70	72	70
query38	2934	2751	2756	2751
query39	854	798	777	777
query40	273	126	127	126
query41	46	44	49	44
query42	102	98	95	95
query43	576	552	567	552
query44	1205	722	739	722
query45	179	160	158	158
query46	1073	734	735	734
query47	1822	1769	1763	1763
query48	367	293	286	286
query49	1157	386	414	386
query50	773	380	381	380
query51	6811	6877	6903	6877
query52	102	88	90	88
query53	359	284	289	284
query54	1005	424	420	420
query55	73	75	72	72
query56	264	237	242	237
query57	1126	1030	1063	1030
query58	220	227	207	207
query59	3434	3187	3155	3155
query60	278	251	251	251
query61	110	90	90	90
query62	654	459	467	459
query63	313	291	296	291
query64	9807	2239	1712	1712
query65	3194	3090	3102	3090
query66	1290	325	340	325
query67	15269	14938	14913	14913
query68	4568	541	547	541
query69	451	265	259	259
query70	1226	1188	1169	1169
query71	419	263	320	263
query72	7901	5622	5780	5622
query73	728	321	324	321
query74	5979	5604	5609	5604
query75	3323	2633	2660	2633
query76	2908	1040	992	992
query77	408	262	269	262
query78	10403	9803	9920	9803
query79	2486	508	516	508
query80	1891	454	449	449
query81	549	247	247	247
query82	704	98	99	98
query83	295	174	176	174
query84	269	87	84	84
query85	2044	332	267	267
query86	492	298	323	298
query87	3290	3087	3109	3087
query88	4319	2359	2331	2331
query89	496	396	386	386
query90	2003	189	197	189
query91	129	98	97	97
query92	60	48	52	48
query93	2539	525	502	502
query94	1229	189	190	189
query95	406	320	314	314
query96	598	268	265	265
query97	3264	3003	3025	3003
query98	243	224	216	216
query99	1207	844	835	835
Total cold run time: 276972 ms
Total hot run time: 173369 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.06
query4	1.67	0.08	0.07
query5	0.48	0.50	0.51
query6	1.14	0.71	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.49
query10	0.55	0.56	0.53
query11	0.16	0.12	0.11
query12	0.14	0.11	0.11
query13	0.59	0.60	0.60
query14	0.78	0.76	0.77
query15	0.84	0.81	0.81
query16	0.36	0.36	0.35
query17	0.97	1.01	0.95
query18	0.22	0.23	0.24
query19	1.76	1.65	1.68
query20	0.02	0.01	0.01
query21	15.54	0.70	0.66
query22	4.26	6.99	2.37
query23	18.25	1.42	1.32
query24	1.48	0.39	0.20
query25	0.13	0.09	0.08
query26	0.26	0.16	0.17
query27	0.08	0.07	0.08
query28	13.30	1.00	0.99
query29	13.21	3.30	3.29
query30	0.24	0.07	0.06
query31	2.86	0.37	0.38
query32	3.29	0.47	0.47
query33	2.84	2.87	2.89
query34	17.02	4.43	4.40
query35	4.64	4.50	4.48
query36	0.65	0.46	0.45
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 30.85 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41036 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

------ Round 1 ----------------------------------
q1	17848	4624	4463	4463
q2	3011	196	200	196
q3	10736	1282	1223	1223
q4	10522	846	746	746
q5	7837	2727	2731	2727
q6	221	139	142	139
q7	991	612	604	604
q8	9447	2147	2136	2136
q9	9028	6506	6492	6492
q10	8955	3733	3773	3733
q11	476	247	242	242
q12	419	236	230	230
q13	17778	3000	2985	2985
q14	275	221	228	221
q15	510	469	481	469
q16	515	383	393	383
q17	979	767	668	668
q18	8123	7517	7385	7385
q19	7220	1536	1470	1470
q20	653	326	322	322
q21	4991	3856	4052	3856
q22	411	354	346	346
Total cold run time: 120946 ms
Total hot run time: 41036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4415	4259	4303	4259
q2	392	260	267	260
q3	2974	2776	2705	2705
q4	1887	1650	1598	1598
q5	5292	5360	5467	5360
q6	217	127	130	127
q7	2172	1745	1764	1745
q8	3226	3359	3342	3342
q9	8356	8392	8396	8392
q10	3931	3648	3676	3648
q11	574	482	495	482
q12	768	591	611	591
q13	17522	2985	3026	2985
q14	293	270	261	261
q15	510	496	468	468
q16	470	440	436	436
q17	1785	1498	1468	1468
q18	7724	7682	7456	7456
q19	1708	1448	1545	1448
q20	1966	1784	1796	1784
q21	8748	4655	4656	4655
q22	619	542	551	542
Total cold run time: 75549 ms
Total hot run time: 54012 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170049 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

query1	945	404	383	383
query2	6473	2415	2366	2366
query3	6874	210	209	209
query4	19583	17388	17437	17388
query5	4150	469	460	460
query6	249	167	160	160
query7	4599	306	291	291
query8	310	282	293	282
query9	8551	2431	2398	2398
query10	441	296	281	281
query11	11038	10020	10159	10020
query12	142	94	86	86
query13	1657	376	365	365
query14	10020	6871	6936	6871
query15	236	190	186	186
query16	7638	277	272	272
query17	1389	529	528	528
query18	1967	268	270	268
query19	219	150	156	150
query20	99	86	85	85
query21	204	128	130	128
query22	4272	3938	3918	3918
query23	33513	32920	33111	32920
query24	12343	2850	2810	2810
query25	657	358	359	358
query26	1777	160	158	158
query27	3036	325	325	325
query28	7629	2067	2071	2067
query29	1155	602	600	600
query30	290	147	151	147
query31	937	745	733	733
query32	92	53	55	53
query33	784	274	282	274
query34	1004	473	479	473
query35	727	646	612	612
query36	1096	942	919	919
query37	209	72	70	70
query38	2859	2739	2736	2736
query39	882	788	767	767
query40	279	125	124	124
query41	53	53	52	52
query42	116	100	98	98
query43	603	548	560	548
query44	1222	732	749	732
query45	200	168	187	168
query46	1098	729	721	721
query47	1868	1793	1761	1761
query48	393	292	303	292
query49	1249	409	414	409
query50	789	392	379	379
query51	6868	6885	6779	6779
query52	108	91	87	87
query53	355	288	287	287
query54	952	458	448	448
query55	75	71	74	71
query56	275	270	258	258
query57	1125	1071	1022	1022
query58	249	280	242	242
query59	3414	3358	3241	3241
query60	295	278	273	273
query61	92	88	89	88
query62	653	439	452	439
query63	321	295	288	288
query64	9892	2178	1729	1729
query65	3188	3127	3143	3127
query66	1367	334	328	328
query67	15637	15219	14979	14979
query68	4569	537	534	534
query69	453	319	325	319
query70	1192	1141	1053	1053
query71	401	282	286	282
query72	7167	2882	2710	2710
query73	748	415	323	323
query74	5894	5566	5540	5540
query75	3344	2695	2664	2664
query76	2305	930	871	871
query77	443	303	292	292
query78	10285	9755	9756	9755
query79	3167	508	506	506
query80	2052	479	452	452
query81	579	228	222	222
query82	780	111	104	104
query83	279	174	178	174
query84	273	94	89	89
query85	2078	268	263	263
query86	519	331	315	315
query87	3286	3069	3070	3069
query88	3948	2359	2338	2338
query89	504	382	386	382
query90	1835	193	192	192
query91	131	144	95	95
query92	58	49	50	49
query93	4349	525	520	520
query94	1275	196	187	187
query95	420	322	315	315
query96	603	269	267	267
query97	3217	3021	3047	3021
query98	254	218	212	212
query99	1224	847	851	847
Total cold run time: 280414 ms
Total hot run time: 170049 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b142db98e5ad3741b5e19699e642f3d3785dfa33, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.06
query5	0.50	0.49	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.52	0.48	0.49
query10	0.54	0.54	0.53
query11	0.15	0.12	0.11
query12	0.16	0.12	0.12
query13	0.59	0.59	0.60
query14	0.76	0.76	0.78
query15	0.84	0.81	0.82
query16	0.37	0.37	0.38
query17	0.97	0.95	0.99
query18	0.22	0.25	0.24
query19	1.79	1.81	1.76
query20	0.02	0.01	0.01
query21	15.56	0.68	0.64
query22	4.08	7.58	2.16
query23	18.32	1.43	1.35
query24	1.87	0.22	0.22
query25	0.15	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.07
query28	13.28	1.03	1.01
query29	13.18	3.30	3.27
query30	0.24	0.06	0.06
query31	2.87	0.38	0.40
query32	3.26	0.48	0.48
query33	2.87	2.89	2.87
query34	17.27	4.36	4.37
query35	4.48	4.46	4.51
query36	0.65	0.46	0.49
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.82 s
Total hot run time: 30.81 s

@CalvinKirs
Copy link
Member Author

run buildall

morningman
morningman approved these changes Jun 11, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40137 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fc950e9ff7f1531b176c9b5edeefcf9070db31ee, data reload: false

------ Round 1 ----------------------------------
q1	17612	4297	4308	4297
q2	2028	202	195	195
q3	10435	1124	1078	1078
q4	10177	785	728	728
q5	7522	2642	2635	2635
q6	218	133	132	132
q7	943	612	602	602
q8	9212	2040	2060	2040
q9	8948	6463	6441	6441
q10	8956	3672	3725	3672
q11	456	229	232	229
q12	478	238	218	218
q13	17774	2982	2979	2979
q14	275	220	225	220
q15	505	470	472	470
q16	526	373	370	370
q17	958	685	686	685
q18	8156	7846	7776	7776
q19	7931	1443	1430	1430
q20	730	332	320	320
q21	4999	3279	4080	3279
q22	415	346	341	341
Total cold run time: 119254 ms
Total hot run time: 40137 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4657	4422	4444	4422
q2	359	245	255	245
q3	2962	2866	2873	2866
q4	1933	1633	1645	1633
q5	5495	5523	5511	5511
q6	228	127	129	127
q7	2108	1788	1868	1788
q8	3220	3385	3389	3385
q9	8644	8617	8719	8617
q10	4098	3814	3893	3814
q11	584	491	494	491
q12	848	669	629	629
q13	15924	2991	3114	2991
q14	303	276	285	276
q15	518	479	486	479
q16	496	413	436	413
q17	1813	1513	1489	1489
q18	7990	7680	7724	7680
q19	1753	1475	1682	1475
q20	2746	1876	1881	1876
q21	4953	4797	4957	4797
q22	609	571	545	545
Total cold run time: 72241 ms
Total hot run time: 55549 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172424 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fc950e9ff7f1531b176c9b5edeefcf9070db31ee, data reload: false

query1	805	393	379	379
query2	6329	2392	2495	2392
query3	6406	202	210	202
query4	21112	17445	17267	17267
query5	3774	468	455	455
query6	250	157	164	157
query7	4469	298	296	296
query8	316	291	287	287
query9	8467	2368	2368	2368
query10	549	291	270	270
query11	10499	10046	10030	10030
query12	134	86	84	84
query13	1607	369	360	360
query14	10036	6804	6838	6804
query15	239	189	185	185
query16	7496	270	260	260
query17	1398	524	513	513
query18	1939	271	270	270
query19	193	148	150	148
query20	87	82	81	81
query21	209	133	126	126
query22	4528	4262	4115	4115
query23	33613	33614	33533	33533
query24	11291	2831	2848	2831
query25	639	380	371	371
query26	1182	150	151	150
query27	2392	313	314	313
query28	6948	2066	2042	2042
query29	905	621	592	592
query30	232	149	150	149
query31	954	714	720	714
query32	99	51	57	51
query33	748	274	269	269
query34	925	461	458	458
query35	741	629	655	629
query36	1087	937	951	937
query37	136	69	66	66
query38	2848	2739	2723	2723
query39	862	777	818	777
query40	205	124	122	122
query41	53	51	49	49
query42	118	93	98	93
query43	560	560	541	541
query44	1215	721	716	716
query45	199	160	161	160
query46	1074	712	719	712
query47	1850	1774	1779	1774
query48	375	294	302	294
query49	854	404	406	404
query50	764	384	375	375
query51	6775	6636	6684	6636
query52	96	97	95	95
query53	360	290	282	282
query54	877	448	446	446
query55	75	98	75	75
query56	268	262	255	255
query57	1116	1029	1076	1029
query58	252	245	234	234
query59	3508	3214	3012	3012
query60	299	277	265	265
query61	88	87	91	87
query62	627	448	446	446
query63	310	289	293	289
query64	8834	2208	1706	1706
query65	3192	3102	3095	3095
query66	862	324	330	324
query67	15431	15152	14922	14922
query68	4480	533	534	533
query69	560	455	401	401
query70	1171	1045	1038	1038
query71	415	271	286	271
query72	6974	5498	5346	5346
query73	753	318	325	318
query74	5811	5513	5444	5444
query75	3324	2625	2601	2601
query76	2321	960	902	902
query77	430	297	302	297
query78	10362	9778	9673	9673
query79	2052	513	519	513
query80	1184	460	447	447
query81	592	215	215	215
query82	830	100	97	97
query83	244	159	159	159
query84	245	84	82	82
query85	1979	276	263	263
query86	482	326	319	319
query87	3264	3075	3086	3075
query88	4250	2383	2363	2363
query89	468	382	373	373
query90	1889	188	186	186
query91	130	96	96	96
query92	60	47	51	47
query93	2314	491	501	491
query94	1276	258	182	182
query95	393	314	308	308
query96	584	277	263	263
query97	3205	3008	3042	3008
query98	219	205	197	197
query99	1161	858	824	824
Total cold run time: 270103 ms
Total hot run time: 172424 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fc950e9ff7f1531b176c9b5edeefcf9070db31ee, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.49	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.56
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.59	0.58	0.60
query14	0.76	0.77	0.79
query15	0.83	0.80	0.81
query16	0.36	0.36	0.37
query17	0.97	0.94	1.04
query18	0.21	0.24	0.24
query19	1.84	1.77	1.76
query20	0.01	0.02	0.01
query21	15.40	0.66	0.64
query22	4.12	7.01	2.13
query23	18.37	1.44	1.20
query24	2.12	0.21	0.23
query25	0.16	0.09	0.09
query26	0.28	0.17	0.17
query27	0.08	0.09	0.08
query28	13.26	1.01	1.01
query29	12.63	3.28	3.22
query30	0.26	0.06	0.06
query31	2.85	0.39	0.38
query32	3.30	0.49	0.46
query33	2.94	2.91	2.86
query34	17.30	4.46	4.41
query35	4.47	4.54	4.44
query36	0.65	0.45	0.45
query37	0.18	0.15	0.15
query38	0.16	0.13	0.14
query39	0.05	0.03	0.03
query40	0.18	0.14	0.15
query41	0.11	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.57 s

JNSimba
JNSimba approved these changes Jun 11, 2024
View reviewed changes
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CalvinKirs CalvinKirs merged commit e0e6495 into apache:master Jun 11, 2024
27 of 29 checks passed
@CalvinKirs CalvinKirs deleted the master-hadoop branch June 11, 2024 05:58
dataroaring pushed a commit that referenced this pull request Jun 13, 2024
@CalvinKirs @dataroaring
[minor](dependencies)binding a third-party Hadoop component to versio…
72e1092 
…n 1.2.0 (#35093)

## Proposed changes
binding a third-party Hadoop component to version 1.2.0


<!--Describe your changes.-->

hadoop-shaded-protobuf and hadoop-shaded-guava are older components, and
the components such as guava they depend on have some known CVE issues,
so they were upgraded to 1.2.0

remove unless hadoop yarn and htrace jars
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JNSimba JNSimba JNSimba approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@CalvinKirs @doris-robot @morningman @JNSimba @gavinchou