Skip to content

[fix](compile) auto-regenerate hadoop-deps/lib when missing due to Ma…#60964

Merged
hello-stephen merged 2 commits intoapache:masterfrom
hello-stephen:0302-fix
Mar 3, 2026
Merged

[fix](compile) auto-regenerate hadoop-deps/lib when missing due to Ma…#60964
hello-stephen merged 2 commits intoapache:masterfrom
hello-stephen:0302-fix

Conversation

@hello-stephen
Copy link
Contributor

…ven cache

Problem:

  • When Maven build cache is hit, the dependency:copy-dependencies plugin
    execution is skipped
  • This causes target/lib directory to be missing in hadoop-deps module
  • Results in an incomplete BE output package without Hadoop dependencies

Solution:

  • Add automatic regeneration logic in build.sh copy phase
  • When lib directory is missing and BUILD_BE_JAVA_EXTENSIONS=1,
    execute 'mvn dependency:copy-dependencies' to regenerate it
  • Preserves Maven cache benefits while ensuring complete output

Impact:

  • Fixes intermittent missing hadoop_hdfs jars in BE output
  • No impact on normal build flow (only triggers when needed)
  • Maintains build cache acceleration for other plugins

Related PR: #60819

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 2, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28897 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c572d3320a4046b3ffa2ccba36beaae8d063214, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17631	4549	4369	4369
q2	q3	10643	794	549	549
q4	4678	360	255	255
q5	7539	1191	1032	1032
q6	182	177	147	147
q7	797	855	686	686
q8	9300	1458	1339	1339
q9	4828	4723	4709	4709
q10	6789	1854	1646	1646
q11	467	257	239	239
q12	680	565	459	459
q13	17757	4247	3418	3418
q14	236	239	208	208
q15	911	806	797	797
q16	727	726	682	682
q17	731	863	412	412
q18	5927	5453	5177	5177
q19	1194	989	628	628
q20	522	487	387	387
q21	4878	1925	1475	1475
q22	361	295	283	283
Total cold run time: 96778 ms
Total hot run time: 28897 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4651	4642	4537	4537
q2	q3	1811	2188	1714	1714
q4	848	1197	756	756
q5	4048	4339	4318	4318
q6	185	171	139	139
q7	1797	1642	1586	1586
q8	2468	2849	2541	2541
q9	7557	7528	7324	7324
q10	2684	2823	2421	2421
q11	514	430	422	422
q12	491	571	432	432
q13	3946	4446	3681	3681
q14	289	300	287	287
q15	822	805	790	790
q16	705	758	689	689
q17	1148	1473	1336	1336
q18	7160	6693	6673	6673
q19	892	1078	967	967
q20	2102	2168	2012	2012
q21	4022	3511	3342	3342
q22	491	430	377	377
Total cold run time: 48631 ms
Total hot run time: 46344 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184030 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c572d3320a4046b3ffa2ccba36beaae8d063214, data reload: false

query5	4856	640	515	515
query6	338	245	206	206
query7	4220	468	283	283
query8	325	242	233	233
query9	8704	2753	2760	2753
query10	536	375	356	356
query11	17024	16898	16651	16651
query12	181	125	123	123
query13	1253	470	360	360
query14	6435	3176	2888	2888
query14_1	2823	2753	2782	2753
query15	197	190	178	178
query16	961	457	451	451
query17	1065	693	589	589
query18	2595	434	335	335
query19	206	207	181	181
query20	141	131	123	123
query21	223	143	121	121
query22	4803	6134	5585	5585
query23	17768	17226	17002	17002
query23_1	16953	17215	16651	16651
query24	7315	1708	1244	1244
query24_1	1249	1237	1231	1231
query25	570	482	441	441
query26	1251	262	150	150
query27	2771	474	298	298
query28	4508	1893	1875	1875
query29	833	584	486	486
query30	323	252	215	215
query31	869	744	667	667
query32	81	77	75	75
query33	534	344	301	301
query34	944	912	571	571
query35	635	674	605	605
query36	1099	1122	992	992
query37	132	98	88	88
query38	2959	2920	2818	2818
query39	885	864	849	849
query39_1	821	840	833	833
query40	236	158	139	139
query41	67	64	66	64
query42	109	105	107	105
query43	371	377	348	348
query44	
query45	203	195	186	186
query46	879	993	612	612
query47	2134	2147	2057	2057
query48	301	322	239	239
query49	645	472	394	394
query50	677	283	213	213
query51	4148	4103	4127	4103
query52	104	105	96	96
query53	291	340	291	291
query54	295	270	254	254
query55	94	83	84	83
query56	314	311	321	311
query57	1361	1335	1268	1268
query58	290	282	275	275
query59	2535	2700	2485	2485
query60	343	333	310	310
query61	147	138	147	138
query62	633	586	543	543
query63	318	271	279	271
query64	4870	1252	990	990
query65	
query66	1417	464	409	409
query67	16378	16478	16319	16319
query68	
query69	392	308	285	285
query70	997	990	956	956
query71	344	305	297	297
query72	2815	2660	2337	2337
query73	531	543	316	316
query74	10002	9959	9724	9724
query75	2830	2757	2457	2457
query76	2317	1029	669	669
query77	353	389	321	321
query78	11260	11528	10689	10689
query79	1159	816	594	594
query80	1347	627	539	539
query81	558	280	258	258
query82	990	152	116	116
query83	341	270	242	242
query84	253	122	99	99
query85	1020	465	444	444
query86	413	320	324	320
query87	3093	3116	3011	3011
query88	3555	2672	2683	2672
query89	424	364	356	356
query90	1922	167	169	167
query91	166	158	133	133
query92	78	78	75	75
query93	1077	817	512	512
query94	644	329	288	288
query95	588	399	313	313
query96	646	528	236	236
query97	2467	2495	2415	2415
query98	234	222	228	222
query99	1009	1015	900	900
Total cold run time: 253694 ms
Total hot run time: 184030 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 28748 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c572d3320a4046b3ffa2ccba36beaae8d063214, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17629	4442	4312	4312
q2	q3	10646	752	525	525
q4	4687	344	274	274
q5	7542	1236	1008	1008
q6	170	179	145	145
q7	780	848	668	668
q8	9307	1475	1343	1343
q9	4843	4749	4639	4639
q10	6760	1869	1647	1647
q11	469	255	243	243
q12	708	561	475	475
q13	17761	4217	3421	3421
q14	228	231	223	223
q15	914	796	787	787
q16	721	715	670	670
q17	734	873	397	397
q18	5882	5368	5277	5277
q19	1246	1003	613	613
q20	500	498	391	391
q21	4985	1965	1427	1427
q22	363	292	263	263
Total cold run time: 96875 ms
Total hot run time: 28748 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4650	4610	4517	4517
q2	q3	1779	2208	1765	1765
q4	873	1206	769	769
q5	4069	4371	4314	4314
q6	187	173	141	141
q7	1774	1611	1541	1541
q8	2462	2876	2513	2513
q9	7436	7390	7475	7390
q10	2637	2840	2356	2356
q11	502	426	423	423
q12	511	590	462	462
q13	4030	4415	3553	3553
q14	288	298	286	286
q15	863	855	826	826
q16	707	739	722	722
q17	1141	1797	1294	1294
q18	7040	6861	6597	6597
q19	844	820	915	820
q20	2155	2149	1988	1988
q21	3957	3479	3340	3340
q22	492	425	371	371
Total cold run time: 48397 ms
Total hot run time: 45988 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183190 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c572d3320a4046b3ffa2ccba36beaae8d063214, data reload: false

query5	4799	641	500	500
query6	346	241	219	219
query7	4216	486	278	278
query8	356	250	256	250
query9	8754	2755	2733	2733
query10	525	409	362	362
query11	16739	17230	16823	16823
query12	199	140	132	132
query13	1321	509	368	368
query14	7226	3304	3118	3118
query14_1	2911	2906	2927	2906
query15	215	204	185	185
query16	1024	516	450	450
query17	1096	774	653	653
query18	2726	435	345	345
query19	214	207	183	183
query20	134	127	126	126
query21	215	138	120	120
query22	5122	5095	4942	4942
query23	17196	16826	16615	16615
query23_1	16735	16796	16778	16778
query24	7141	1602	1234	1234
query24_1	1232	1262	1222	1222
query25	598	449	409	409
query26	1234	256	144	144
query27	2782	469	281	281
query28	4443	1861	1870	1861
query29	775	547	457	457
query30	309	243	209	209
query31	865	719	657	657
query32	84	66	73	66
query33	515	323	277	277
query34	897	901	561	561
query35	624	670	590	590
query36	1076	1140	981	981
query37	133	98	85	85
query38	2931	2933	2831	2831
query39	886	883	875	875
query39_1	817	845	821	821
query40	230	150	136	136
query41	62	61	58	58
query42	103	100	106	100
query43	383	384	337	337
query44	
query45	197	189	179	179
query46	875	972	597	597
query47	2157	2163	2074	2074
query48	313	310	248	248
query49	648	469	376	376
query50	675	284	206	206
query51	4237	4101	4046	4046
query52	103	106	98	98
query53	288	344	277	277
query54	286	267	295	267
query55	85	82	80	80
query56	306	308	294	294
query57	1362	1344	1287	1287
query58	293	276	270	270
query59	2598	2679	2537	2537
query60	324	338	334	334
query61	154	151	166	151
query62	613	588	540	540
query63	306	290	272	272
query64	4842	1272	992	992
query65	
query66	1373	448	369	369
query67	16439	16224	16249	16224
query68	
query69	382	310	281	281
query70	978	963	965	963
query71	339	300	295	295
query72	2756	2693	2386	2386
query73	529	544	323	323
query74	9928	9899	9722	9722
query75	2833	2737	2448	2448
query76	2300	1027	655	655
query77	358	404	305	305
query78	11282	11439	10669	10669
query79	3125	775	595	595
query80	1811	624	538	538
query81	615	279	238	238
query82	1014	150	121	121
query83	346	260	244	244
query84	257	120	93	93
query85	969	484	432	432
query86	498	311	291	291
query87	3114	3081	2982	2982
query88	3541	2684	2647	2647
query89	423	367	339	339
query90	2175	171	177	171
query91	177	160	136	136
query92	89	80	69	69
query93	2240	849	517	517
query94	652	328	319	319
query95	598	339	373	339
query96	628	515	228	228
query97	2469	2515	2433	2433
query98	233	219	217	217
query99	926	981	940	940
Total cold run time: 257420 ms
Total hot run time: 183190 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 3, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2026

PR approved by anyone and no changes requested.

Copy link
Collaborator

@shuke987 shuke987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen hello-stephen merged commit c6ee9cf into apache:master Mar 3, 2026
31 of 32 checks passed
@hello-stephen hello-stephen deleted the 0302-fix branch March 3, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants