Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor](spark load)(cherry-pick)update spark version for spark load to resolve cve problem #33085

Merged
merged 1 commit into from
Apr 1, 2024

Conversation

gnehil
Copy link
Contributor

@gnehil gnehil commented Apr 1, 2024

Proposed changes

Issue Number: close #xxx

(cherry picked from commit #30368)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gnehil gnehil changed the title [refactor](spark load)()update spark version for spark load to resolve cve problem [refactor](spark load)(cherry-pick)update spark version for spark load to resolve cve problem Apr 1, 2024
@CalvinKirs
Copy link
Member

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49748 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86ab963e06abac2ad434624f4b9682850a80bf5a, data reload: false

------ Round 1 ----------------------------------
q1	17920	4407	4364	4364
q2	2031	150	144	144
q3	10457	1883	1912	1883
q4	10328	1226	1297	1226
q5	8880	3941	3990	3941
q6	233	125	121	121
q7	2044	1593	1609	1593
q8	9280	2698	2704	2698
q9	11124	10689	10463	10463
q10	8602	3511	3472	3472
q11	421	241	232	232
q12	462	296	304	296
q13	18332	3941	4039	3941
q14	367	323	320	320
q15	511	459	452	452
q16	707	587	578	578
q17	1139	971	980	971
q18	7286	6825	6775	6775
q19	1684	1567	1526	1526
q20	549	328	304	304
q21	4419	4111	4050	4050
q22	487	399	398	398
Total cold run time: 117263 ms
Total hot run time: 49748 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4281	4289	4281
q2	316	223	221	221
q3	4202	4169	4163	4163
q4	2756	2722	2731	2722
q5	7250	7193	7203	7193
q6	233	119	119	119
q7	3226	2826	2824	2824
q8	4386	4445	4438	4438
q9	17669	17403	17222	17222
q10	4235	4281	4243	4243
q11	759	686	660	660
q12	1013	864	850	850
q13	6594	3760	3726	3726
q14	452	435	425	425
q15	501	456	463	456
q16	754	701	715	701
q17	3880	3851	3813	3813
q18	8957	8856	8828	8828
q19	1720	1694	1632	1632
q20	2372	2144	2092	2092
q21	8491	8610	8537	8537
q22	1068	969	952	952
Total cold run time: 85159 ms
Total hot run time: 80098 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201057 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86ab963e06abac2ad434624f4b9682850a80bf5a, data reload: false

query1	933	389	373	373
query2	6523	2044	2081	2044
query3	6918	199	195	195
query4	20468	18008	17993	17993
query5	19718	6461	6501	6461
query6	274	218	228	218
query7	4152	293	297	293
query8	253	222	260	222
query9	3127	2722	2649	2649
query10	414	278	298	278
query11	11313	10664	10796	10664
query12	122	76	74	74
query13	5584	639	628	628
query14	17665	13518	13243	13243
query15	361	238	241	238
query16	6447	273	255	255
query17	1728	1446	881	881
query18	2319	407	393	393
query19	203	150	156	150
query20	73	75	75	75
query21	189	89	88	88
query22	5223	5033	4976	4976
query23	32303	31839	31867	31839
query24	6864	6454	6467	6454
query25	521	437	408	408
query26	523	155	153	153
query27	1881	291	291	291
query28	6159	2236	2225	2225
query29	2832	2714	2634	2634
query30	242	159	164	159
query31	915	728	720	720
query32	64	61	59	59
query33	395	240	232	232
query34	848	459	479	459
query35	1117	901	930	901
query36	1341	1172	1178	1172
query37	85	61	58	58
query38	3080	2940	2891	2891
query39	1354	1314	1311	1311
query40	201	94	92	92
query41	37	32	31	31
query42	82	75	80	75
query43	624	634	601	601
query44	1124	722	720	720
query45	240	222	227	222
query46	1230	970	980	970
query47	2043	1626	1892	1626
query48	969	669	666	666
query49	611	372	369	369
query50	864	608	608	608
query51	4691	4593	4619	4593
query52	83	87	79	79
query53	436	319	314	314
query54	2682	2439	2442	2439
query55	84	73	86	73
query56	215	210	203	203
query57	1260	1065	1071	1065
query58	216	197	181	181
query59	3505	3288	3380	3288
query60	205	188	184	184
query61	84	84	85	84
query62	819	452	555	452
query63	464	335	332	332
query64	2337	1448	1291	1291
query65	3625	3532	3585	3532
query66	749	362	362	362
query67	17001	15486	17624	15486
query68	8484	664	669	664
query69	560	336	319	319
query70	1497	1343	1518	1343
query71	410	305	309	305
query72	6557	3379	3453	3379
query73	716	324	317	317
query74	6261	5906	5880	5880
query75	4651	3722	3777	3722
query76	4828	1160	1195	1160
query77	683	244	251	244
query78	12659	11764	11813	11764
query79	6917	629	642	629
query80	1061	383	390	383
query81	489	232	233	232
query82	1503	110	90	90
query83	169	126	125	125
query84	253	68	70	68
query85	874	288	275	275
query86	334	297	304	297
query87	3250	3023	2990	2990
query88	4893	2322	2293	2293
query89	334	292	284	284
query90	1922	211	212	211
query91	149	115	112	112
query92	57	52	55	52
query93	3212	576	555	555
query94	749	206	206	206
query95	1125	1069	1074	1069
query96	650	324	322	322
query97	6547	6441	6538	6441
query98	189	165	169	165
query99	2951	981	866	866
Total cold run time: 305946 ms
Total hot run time: 201057 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86ab963e06abac2ad434624f4b9682850a80bf5a, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.06
query4	1.79	0.06	0.06
query5	0.53	0.53	0.52
query6	1.30	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.47	0.48
query10	0.53	0.55	0.53
query11	0.13	0.08	0.09
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.80	0.81	0.78
query15	0.80	0.75	0.77
query16	0.37	0.39	0.36
query17	1.01	0.95	1.03
query18	0.21	0.27	0.24
query19	1.92	1.85	1.87
query20	0.02	0.01	0.01
query21	15.52	0.58	0.55
query22	2.11	1.99	1.60
query23	17.10	0.94	0.86
query24	5.56	4.45	0.97
query25	0.43	0.09	0.05
query26	0.99	0.16	0.18
query27	0.05	0.04	0.03
query28	4.24	0.81	0.74
query29	12.67	2.27	2.36
query30	0.59	0.53	0.52
query31	2.82	0.38	0.37
query32	3.39	0.49	0.48
query33	3.06	3.02	3.08
query34	15.27	4.81	4.79
query35	4.86	4.86	4.87
query36	1.05	1.02	1.01
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 101.13 s
Total hot run time: 30.58 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 86ab963e06abac2ad434624f4b9682850a80bf5a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s

@CalvinKirs CalvinKirs merged commit 808c9cd into apache:branch-2.0 Apr 1, 2024
27 of 28 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants