Skip to content

[fix](cloud): fix parent directory doesn't exist in HdfsFileWriter#33985

Merged
spaces-X merged 1 commit intoapache:masterfrom
spaces-X:master
Apr 25, 2024
Merged

[fix](cloud): fix parent directory doesn't exist in HdfsFileWriter#33985
spaces-X merged 1 commit intoapache:masterfrom
spaces-X:master

Conversation

@spaces-X
Copy link
Contributor

@spaces-X spaces-X commented Apr 22, 2024

Proposed changes

Issue Number: close #33983

create parent dir in HdfsFileWriter

introduced from #33796

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@spaces-X
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8913/25209)
Line Coverage: 27.08% (73290/270605)
Region Coverage: 26.22% (37851/144372)
Branch Coverage: 23.03% (19277/83700)
Coverage Report: http://coverage.selectdb-in.cc/coverage/be5479645c5ed9faa59ce4f3147f4dabb21bcf19_be5479645c5ed9faa59ce4f3147f4dabb21bcf19/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38057 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be5479645c5ed9faa59ce4f3147f4dabb21bcf19, data reload: false

------ Round 1 ----------------------------------
q1	17591	4294	4198	4198
q2	2035	189	179	179
q3	10457	1175	1180	1175
q4	10200	764	792	764
q5	7492	2685	2622	2622
q6	214	131	130	130
q7	1015	596	577	577
q8	9214	2032	2002	2002
q9	7240	6556	6509	6509
q10	8602	3513	3492	3492
q11	457	227	222	222
q12	466	223	213	213
q13	17765	2904	2932	2904
q14	267	233	234	233
q15	521	478	475	475
q16	514	382	375	375
q17	943	640	669	640
q18	7338	6757	6562	6562
q19	4491	1446	1475	1446
q20	643	321	295	295
q21	3482	2741	2826	2741
q22	356	313	303	303
Total cold run time: 111303 ms
Total hot run time: 38057 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4174	4252	4174
q2	371	270	263	263
q3	2981	2748	2775	2748
q4	1820	1538	1547	1538
q5	5312	5309	5280	5280
q6	209	124	122	122
q7	2225	1825	1824	1824
q8	3188	3328	3321	3321
q9	8540	8509	8527	8509
q10	3993	3846	3852	3846
q11	587	485	524	485
q12	823	624	632	624
q13	16428	3293	3193	3193
q14	309	294	302	294
q15	523	478	476	476
q16	487	447	436	436
q17	1825	1498	1486	1486
q18	8141	7828	7989	7828
q19	1628	1504	1544	1504
q20	2043	1885	1828	1828
q21	8899	4929	4914	4914
q22	553	482	468	468
Total cold run time: 75257 ms
Total hot run time: 55161 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185073 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be5479645c5ed9faa59ce4f3147f4dabb21bcf19, data reload: false

query1	885	369	371	369
query2	6228	2593	2375	2375
query3	6659	195	197	195
query4	23702	21451	21169	21169
query5	4130	399	406	399
query6	267	174	181	174
query7	4588	277	286	277
query8	246	202	180	180
query9	8490	2320	2292	2292
query10	420	240	251	240
query11	14628	14231	14205	14205
query12	130	89	82	82
query13	1628	350	350	350
query14	9513	7917	7617	7617
query15	273	180	177	177
query16	8154	257	250	250
query17	1939	584	564	564
query18	2106	284	285	284
query19	332	153	155	153
query20	91	87	84	84
query21	199	126	126	126
query22	5011	4822	4840	4822
query23	33642	32930	33376	32930
query24	10794	3013	3130	3013
query25	603	404	410	404
query26	717	167	155	155
query27	2343	365	374	365
query28	6032	2085	2040	2040
query29	883	633	629	629
query30	321	180	173	173
query31	997	778	775	775
query32	95	60	61	60
query33	700	271	317	271
query34	878	487	516	487
query35	834	731	703	703
query36	1078	925	904	904
query37	117	69	71	69
query38	3502	3389	3331	3331
query39	1595	1559	1603	1559
query40	173	134	125	125
query41	43	42	48	42
query42	105	99	100	99
query43	599	525	547	525
query44	1098	736	726	726
query45	276	288	284	284
query46	1081	737	743	737
query47	2019	1949	1935	1935
query48	390	289	301	289
query49	839	392	433	392
query50	789	397	397	397
query51	6921	6773	6707	6707
query52	98	84	89	84
query53	345	278	270	270
query54	297	232	215	215
query55	76	70	69	69
query56	236	220	218	218
query57	1235	1150	1134	1134
query58	220	197	200	197
query59	3474	3085	3087	3085
query60	251	231	232	231
query61	90	85	84	84
query62	631	452	446	446
query63	311	290	290	290
query64	4534	3890	3774	3774
query65	3084	3006	2993	2993
query66	745	321	320	320
query67	15174	15165	15261	15165
query68	6807	546	530	530
query69	564	293	303	293
query70	1244	1165	1068	1068
query71	1458	1259	1258	1258
query72	6559	2615	2421	2421
query73	723	314	319	314
query74	6882	6335	6310	6310
query75	3930	2631	2628	2628
query76	4821	1103	1110	1103
query77	599	272	263	263
query78	10994	10316	10244	10244
query79	8196	516	508	508
query80	1474	488	436	436
query81	516	243	241	241
query82	864	90	94	90
query83	190	164	168	164
query84	269	81	84	81
query85	1218	255	257	255
query86	460	298	292	292
query87	3465	3271	3302	3271
query88	5046	2311	2330	2311
query89	549	373	364	364
query90	1871	180	184	180
query91	124	95	97	95
query92	55	45	49	45
query93	6586	514	510	510
query94	1017	175	177	175
query95	382	287	300	287
query96	606	261	263	261
query97	3118	2944	2912	2912
query98	228	216	212	212
query99	1271	866	873	866
Total cold run time: 291970 ms
Total hot run time: 185073 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be5479645c5ed9faa59ce4f3147f4dabb21bcf19, data reload: false

query1	0.03	0.03	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.10
query5	0.50	0.49	0.50
query6	1.43	0.72	0.70
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.53	0.56	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	0.76	0.75	0.76
query15	0.82	0.80	0.79
query16	0.35	0.37	0.37
query17	1.02	1.00	1.04
query18	0.22	0.22	0.25
query19	1.77	1.68	1.67
query20	0.01	0.01	0.01
query21	15.40	0.66	0.66
query22	4.42	6.20	2.67
query23	18.33	1.40	1.25
query24	1.89	0.24	0.20
query25	0.14	0.09	0.07
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.31	0.99	0.99
query29	12.61	3.24	3.22
query30	0.25	0.06	0.08
query31	2.82	0.36	0.38
query32	3.29	0.45	0.47
query33	2.79	2.78	2.76
query34	17.10	4.44	4.36
query35	4.46	4.53	4.45
query36	0.67	0.48	0.46
query37	0.18	0.14	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.48 s
Total hot run time: 30.85 s

@spaces-X
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8915/25209)
Line Coverage: 27.09% (73311/270611)
Region Coverage: 26.23% (37864/144362)
Branch Coverage: 23.04% (19283/83696)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fab565725f6695ed1735f5b428dfb755c9583765_fab565725f6695ed1735f5b428dfb755c9583765/report/index.html

@spaces-X spaces-X requested a review from platoneko April 25, 2024 03:34
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 25, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@spaces-X spaces-X requested a review from 924060929 April 25, 2024 06:32
@spaces-X spaces-X merged commit 693a485 into apache:master Apr 25, 2024
dataroaring pushed a commit that referenced this pull request Apr 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] parent directory doesn't exist in HdfsFileWriter

4 participants