HADOOP-17594. DistCp: Expose the JobId for applications executing through run method #2786

ayushtkn · 2021-03-18T12:27:54Z

https://issues.apache.org/jira/browse/HADOOP-17594

…ough run method

ayushtkn · 2021-03-18T12:44:01Z

Ran the S3A Test:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 182.667 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0

AWS_REGION=ap-south-1

hadoop-yetus · 2021-03-18T14:15:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 4s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 21s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 24s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 44s		trunk passed
+1 💚	shadedclient	16m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 24s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 16s	/results-checkstyle-hadoop-tools_hadoop-distcp.txt	hadoop-tools/hadoop-distcp: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17)
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 50s		the patch passed
+1 💚	shadedclient	15m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	29m 0s	/patch-unit-hadoop-tools_hadoop-distcp.txt	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		105m 54s

Reason	Tests
Failed junit tests	hadoop.tools.TestDistCpSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/1/artifact/out/Dockerfile
GITHUB PR	#2786
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8c6e10638ad6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `81db9bc`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/1/testReport/
Max. process+thread count	570 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2021-03-18T14:18:27Z

Test failure not related, That is due to timeout, failed here as well
#2773 (comment)
Have increased the timeout in my other PR.

hadoop-yetus · 2021-03-18T18:08:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 34s		trunk passed
+1 💚	compile	0m 33s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 29s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 25s		trunk passed
+1 💚	mvnsite	0m 33s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	13m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 24s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	13m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	15m 50s	/patch-unit-hadoop-tools_hadoop-distcp.txt	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		85m 24s

Reason	Tests
Failed junit tests	hadoop.tools.TestDistCpSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/2/artifact/out/Dockerfile
GITHUB PR	#2786
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e02a759899a8 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bf3817c`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/2/testReport/
Max. process+thread count	537 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM

+1 pending you move the import. I know it's a losing battle (spark has an automated an ruthless enforcer, for us it is too late).

steveloughran · 2021-03-18T21:51:00Z

...hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java

@@ -43,6 +44,7 @@
 import org.apache.hadoop.tools.DistCpOptions;
 import org.apache.hadoop.tools.mapred.CopyMapper;

+import org.apache.hadoop.tools.util.DistCpTestUtils;


move to hadoop block

hadoop-yetus · 2021-03-18T23:57:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 17s		trunk passed
+1 💚	compile	0m 29s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 25s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 21s		trunk passed
+1 💚	mvnsite	0m 31s		trunk passed
+1 💚	javadoc	0m 25s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 44s		trunk passed
+1 💚	shadedclient	16m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 15s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 49s		the patch passed
+1 💚	shadedclient	15m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	27m 53s	/patch-unit-hadoop-tools_hadoop-distcp.txt	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		104m 27s

Reason	Tests
Failed junit tests	hadoop.tools.TestDistCpSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/3/artifact/out/Dockerfile
GITHUB PR	#2786
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4e460266ffa7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b756b3d`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/3/testReport/
Max. process+thread count	557 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2786/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

liuml07

+1 ~~and agree on Steve's comment.~~ UPDATE: latest commit looks good!

…ough run method (#2786). Contributed by Ayush Saxena. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Steve Loughran <stevel@apache.org>

ayushtkn · 2021-03-19T09:10:40Z

Committed all the way to 3.1, Thanx @liuml07 and @steveloughran

…ough run method (apache#2786). Contributed by Ayush Saxena. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Steve Loughran <stevel@apache.org>

HADOOP-17594. DistCp: Expose the JobId for applications executing thr…

81db9bc

…ough run method

Fix Checkstyle.

bf3817c

ayushtkn requested a review from aajisaka March 18, 2021 17:03

steveloughran approved these changes Mar 18, 2021

View reviewed changes

Move Import Above.

b756b3d

liuml07 approved these changes Mar 19, 2021

View reviewed changes

ayushtkn merged commit 4781761 into apache:trunk Mar 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17594. DistCp: Expose the JobId for applications executing through run method #2786

HADOOP-17594. DistCp: Expose the JobId for applications executing through run method #2786

ayushtkn commented Mar 18, 2021

ayushtkn commented Mar 18, 2021

hadoop-yetus commented Mar 18, 2021

ayushtkn commented Mar 18, 2021

hadoop-yetus commented Mar 18, 2021

steveloughran left a comment

steveloughran Mar 18, 2021

hadoop-yetus commented Mar 18, 2021

liuml07 left a comment •

edited

ayushtkn commented Mar 19, 2021

HADOOP-17594. DistCp: Expose the JobId for applications executing through run method #2786

HADOOP-17594. DistCp: Expose the JobId for applications executing through run method #2786

Conversation

ayushtkn commented Mar 18, 2021

ayushtkn commented Mar 18, 2021

hadoop-yetus commented Mar 18, 2021

ayushtkn commented Mar 18, 2021

hadoop-yetus commented Mar 18, 2021

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Mar 18, 2021

Choose a reason for hiding this comment

hadoop-yetus commented Mar 18, 2021

liuml07 left a comment • edited

Choose a reason for hiding this comment

ayushtkn commented Mar 19, 2021

liuml07 left a comment •

edited