Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](multi-catalog)unsupported hive input format should throw an exception and remove useless method. #29087

Merged
merged 1 commit into from
Dec 28, 2023
Merged

[fix](multi-catalog)unsupported hive input format should throw an exception and remove useless method. #29087

merged 1 commit into from
Dec 28, 2023

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Dec 26, 2023

Proposed changes

introduce from: #28644

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@wsjz
Copy link
Contributor Author

wsjz commented Dec 26, 2023

run buildall

@wsjz
Copy link
Contributor Author

wsjz commented Dec 26, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 7cb1be595881e35afa080d23677794e9b0734836, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4769	4477	4485	4477
q2	369	157	157	157
q3	1489	1311	1229	1229
q4	1149	967	968	967
q5	3168	3159	3165	3159
q6	257	141	137	137
q7	1027	505	490	490
q8	2268	2263	2230	2230
q9	6730	6704	6654	6654
q10	3207	3274	3279	3274
q11	333	201	202	201
q12	354	214	214	214
q13	4531	3802	3757	3757
q14	242	217	218	217
q15	581	522	529	522
q16	442	382	390	382
q17	1040	838	644	644
q18	7038	6816	6775	6775
q19	1578	1594	1606	1594
q20	566	318	320	318
q21	3151	2769	2758	2758
q22	371	299	311	299
Total cold run time: 44660 ms
Total hot run time: 40455 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4393	4438	4418	4418
q2	274	171	178	171
q3	3529	3501	3511	3501
q4	2437	2409	2425	2409
q5	5721	5722	5735	5722
q6	249	128	125	125
q7	2404	1918	1865	1865
q8	3610	3614	3616	3614
q9	9066	9006	8939	8939
q10	3915	3999	4012	3999
q11	480	392	382	382
q12	781	612	599	599
q13	4270	3541	3567	3541
q14	292	251	251	251
q15	578	523	517	517
q16	505	434	456	434
q17	1973	1967	1944	1944
q18	8573	8296	8289	8289
q19	1827	1818	1814	1814
q20	2265	1954	1945	1945
q21	6613	6388	6217	6217
q22	549	454	471	454
Total cold run time: 64304 ms
Total hot run time: 61150 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.91 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 20.8 seconds inserted 10000000 Rows, about 480K ops/s
storage size: 17183343774 Bytes

@wsjz
Copy link
Contributor Author

wsjz commented Dec 26, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 3ef86898ed732195cd02c40c1358508d516ba22d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4803	4525	4507	4507
q2	389	165	158	158
q3	1483	1336	1222	1222
q4	1156	991	960	960
q5	3210	3161	3189	3161
q6	254	136	134	134
q7	1031	493	479	479
q8	2267	2269	2276	2269
q9	6778	6721	6726	6721
q10	3224	3298	3287	3287
q11	335	213	203	203
q12	364	207	213	207
q13	4548	3847	3816	3816
q14	242	214	213	213
q15	577	533	534	533
q16	443	392	384	384
q17	1025	774	659	659
q18	7105	6719	6755	6719
q19	1580	1611	1628	1611
q20	538	327	284	284
q21	3269	2728	2811	2728
q22	369	302	314	302
Total cold run time: 44990 ms
Total hot run time: 40557 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4423	4413	4382	4382
q2	270	170	169	169
q3	3519	3527	3510	3510
q4	2441	2421	2425	2421
q5	5732	5716	5757	5716
q6	249	127	129	127
q7	2407	1898	1851	1851
q8	3631	3633	3620	3620
q9	9089	9040	9052	9040
q10	3921	4041	4030	4030
q11	483	360	370	360
q12	786	609	621	609
q13	4296	3576	3573	3573
q14	283	262	256	256
q15	561	531	533	531
q16	495	456	454	454
q17	1989	1960	1961	1960
q18	8761	8210	8172	8172
q19	1833	1817	1836	1817
q20	2266	1966	1951	1951
q21	6633	6255	6271	6255
q22	568	459	464	459
Total cold run time: 64636 ms
Total hot run time: 61263 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.13 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17183640366 Bytes

dutyu
dutyu approved these changes Dec 27, 2023
View reviewed changes
Copy link
Contributor

@dutyu dutyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 28, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

morningman
morningman approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 8becf05 into apache:master Dec 28, 2023
29 of 30 checks passed
@wsjz wsjz mentioned this pull request Dec 28, 2023
Merged
morningman pushed a commit that referenced this pull request Dec 28, 2023
@wsjz
[fix](multi-catalog)unsupported hive input format should throw an exc…
50dba71 
…eption and remove useless method. (#29228)

from: #29087
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@wsjz @HappenLee
[fix](multi-catalog)unsupported hive input format should throw an exc…
beafb79 
…eption and remove useless method (apache#29087)

introduce from: apache#28644
@wsjz wsjz deleted the hive_re branch March 22, 2024 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangstar333 zhangstar333 zhangstar333 approved these changes

@dutyu dutyu dutyu approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@wsjz @doris-robot @morningman @dutyu @zhangstar333 @xiaokang @wm1581066