Skip to content

[fix](iceberg) iceberg hms support hive1 #30985

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Feb 18, 2024
Merged

[fix](iceberg) iceberg hms support hive1 #30985

morningman merged 1 commit into from
Feb 18, 2024

Conversation

@Emor-nj
Copy link
Contributor

@Emor-nj Emor-nj commented Feb 8, 2024

Proposed changes

master #30984

Further comments

iceberg hms support hive1

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Emor-nj Emor-nj changed the title [fix](iceberg) iceberg hms support hive1(#30984) [fix](iceberg) iceberg hms support hive1 Feb 8, 2024
@Emor-nj
Copy link
Contributor Author

Emor-nj commented Feb 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

TPC-H: Total hot run time: 41705 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41b23dbbf77fdc3dd07d1c1d7c039c69c7aa5dba, data reload: false

------ Round 1 ----------------------------------
q1	19163	5150	5071	5071
q2	2813	151	139	139
q3	10881	1014	1006	1006
q4	4654	973	990	973
q5	7690	3211	3275	3211
q6	202	133	134	133
q7	1284	791	784	784
q8	9234	2095	2068	2068
q9	7574	6712	6692	6692
q10	8310	2663	2653	2653
q11	403	215	220	215
q12	762	333	325	325
q13	17982	3683	3676	3676
q14	295	263	259	259
q15	591	513	495	495
q16	471	406	431	406
q17	916	869	865	865
q18	7387	6649	6777	6649
q19	1549	1488	1483	1483
q20	599	354	344	344
q21	6433	3917	3975	3917
q22	866	344	341	341
Total cold run time: 110059 ms
Total hot run time: 41705 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4805	4816	4801	4801
q2	297	183	181	181
q3	3596	3593	3567	3567
q4	2518	2521	2501	2501
q5	5767	5763	5783	5763
q6	211	125	126	125
q7	2256	1672	1692	1672
q8	3041	3091	3135	3091
q9	8742	8715	8744	8715
q10	6897	4249	4266	4249
q11	530	377	364	364
q12	765	529	542	529
q13	4218	3407	3387	3387
q14	279	231	228	228
q15	598	538	503	503
q16	472	430	442	430
q17	1668	1600	1590	1590
q18	8322	7718	7756	7718
q19	1626	1630	1635	1630
q20	2109	1827	1810	1810
q21	6599	6179	6156	6156
q22	562	502	518	502
Total cold run time: 65878 ms
Total hot run time: 59512 ms

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

TPC-DS: Total hot run time: 180372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41b23dbbf77fdc3dd07d1c1d7c039c69c7aa5dba, data reload: false

query1	949	355	346	346
query2	6516	1744	1608	1608
query3	6697	216	203	203
query4	24512	22461	22481	22461
query5	4251	372	380	372
query6	259	173	175	173
query7	4610	296	287	287
query8	247	195	217	195
query9	8450	2817	2808	2808
query10	413	217	229	217
query11	16058	15565	15632	15565
query12	131	86	85	85
query13	1706	414	415	414
query14	8998	7584	7622	7584
query15	216	182	192	182
query16	7088	257	247	247
query17	1345	570	544	544
query18	2093	267	259	259
query19	191	147	144	144
query20	83	79	82	79
query21	185	121	120	120
query22	5017	4881	4950	4881
query23	32593	31651	31621	31621
query24	12700	3397	3394	3394
query25	656	370	358	358
query26	1872	147	157	147
query27	3053	309	309	309
query28	6606	1839	1811	1811
query29	1239	608	628	608
query30	276	139	153	139
query31	962	757	773	757
query32	94	56	55	55
query33	717	231	224	224
query34	1103	483	492	483
query35	966	836	817	817
query36	1011	894	853	853
query37	188	58	64	58
query38	3419	3319	3356	3319
query39	1357	1346	1317	1317
query40	282	105	104	104
query41	37	35	34	34
query42	106	97	98	97
query43	459	432	440	432
query44	1090	683	704	683
query45	202	184	178	178
query46	1051	795	755	755
query47	1774	1561	1590	1561
query48	416	344	347	344
query49	1191	301	294	294
query50	787	399	370	370
query51	5304	5163	5180	5163
query52	106	90	95	90
query53	390	302	295	295
query54	300	224	226	224
query55	82	80	79	79
query56	228	200	195	195
query57	1087	980	994	980
query58	220	192	204	192
query59	2206	2120	2236	2120
query60	245	216	220	216
query61	85	83	81	81
query62	607	391	388	388
query63	318	287	277	277
query64	6356	3032	3145	3032
query65	3302	3253	3249	3249
query66	1392	323	317	317
query67	14584	14280	14307	14280
query68	5037	546	551	546
query69	493	351	365	351
query70	1251	1289	1199	1199
query71	323	264	256	256
query72	6288	2773	2601	2601
query73	697	308	313	308
query74	7097	6751	6634	6634
query75	3200	2544	2536	2536
query76	3026	1161	1203	1161
query77	337	236	228	228
query78	9487	8748	8792	8748
query79	964	490	497	490
query80	533	367	339	339
query81	438	200	208	200
query82	168	82	87	82
query83	140	124	119	119
query84	234	78	77	77
query85	906	336	330	330
query86	298	310	327	310
query87	3658	3470	3376	3376
query88	2767	2281	2276	2276
query89	425	354	366	354
query90	2131	158	168	158
query91	152	126	125	125
query92	51	49	49	49
query93	965	472	488	472
query94	1237	185	183	183
query95	8503	374	8599	374
query96	588	270	262	262
query97	4423	4256	4298	4256
query98	227	196	198	196
query99	1049	767	702	702
Total cold run time: 280127 ms
Total hot run time: 180372 ms

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41b23dbbf77fdc3dd07d1c1d7c039c69c7aa5dba, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.08
query4	1.64	0.09	0.07
query5	0.49	0.49	0.48
query6	1.39	0.61	0.61
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.53	0.45	0.45
query10	0.50	0.48	0.50
query11	0.12	0.09	0.09
query12	0.12	0.10	0.10
query13	0.59	0.60	0.59
query14	0.76	0.78	0.79
query15	0.80	0.80	0.80
query16	0.33	0.33	0.33
query17	0.90	0.86	0.92
query18	0.18	0.18	0.15
query19	1.76	1.70	1.64
query20	0.01	0.00	0.01
query21	15.41	0.68	0.62
query22	3.36	3.81	2.78
query23	17.70	1.05	0.91
query24	1.97	0.78	0.37
query25	0.64	0.07	0.06
query26	0.16	0.14	0.14
query27	0.05	0.05	0.05
query28	11.81	0.84	0.83
query29	12.55	3.35	3.21
query30	0.49	0.49	0.46
query31	2.77	0.36	0.36
query32	3.34	0.47	0.47
query33	3.13	3.17	3.14
query34	15.35	4.49	4.49
query35	4.50	4.48	4.50
query36	1.08	0.95	0.94
query37	0.07	0.06	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.01
query40	0.18	0.15	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.01
Total cold run time: 105.23 s
Total hot run time: 31.17 s

@github-actions
Copy link
Contributor

github-actions bot commented Feb 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 8, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 8, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 41b23dbbf77fdc3dd07d1c1d7c039c69c7aa5dba with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@wm1581066 wm1581066 added the usercase Important user case type label label Feb 18, 2024
morningman
morningman approved these changes Feb 18, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 18737bf into apache:master Feb 18, 2024
morningman pushed a commit to morningman/doris that referenced this pull request Feb 18, 2024
@Emor-nj @morningman
[fix](iceberg) iceberg hms support hive1 (apache#30985)
48b6d83
@morningman morningman mentioned this pull request Feb 18, 2024
Merged
xiaokang pushed a commit that referenced this pull request Feb 18, 2024
@morningman @Emor-nj
[fix](iceberg) iceberg hms support hive1 (#30985) (#31052)
af6145b 
Co-authored-by: wangtao <emor93@foxmail.com>
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
@Emor-nj @mymeiyi
[fix](iceberg) iceberg hms support hive1 (apache#30985)
55a0429
yiguolei pushed a commit that referenced this pull request Feb 19, 2024
@Emor-nj @Doris-Extras
[fix](iceberg) iceberg hms support hive1 (#30985)
a7037df
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman @Emor-nj
[fix](iceberg) iceberg hms support hive1 (apache#30985) (apache#31052)
ed8268c 
Co-authored-by: wangtao <emor93@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Emor-nj @doris-robot @morningman @yiguolei @xiaokang @wm1581066