HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel #919

kai33 · 2019-06-06T15:47:00Z

Copying blocks in parallel (enabled when blocks per chunk > 0) is a great DistCp improvement that can hugely speed up copying big files.
But its checksum validation is skipped, e.g. in RetriableFileCopyCommand.java

// isSplit() here is introduced by HADOOP-11794 and 
// used to indicate if the source data to copy is only a chunk of it 
// (consists of one or more blocks, not all).
if (!source.isSplit()) {
  compareCheckSums(sourceFS, source.getPath(), sourceChecksum,
      targetFS, targetPath);
}

and this could result in checksum/data mismatch without notifying developers/users (e.g. HADOOP-16049).
I'd like to provide a patch to add the checksum validation.

hadoop-yetus · 2019-06-06T16:10:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
-1	mvninstall	51	root in trunk failed.
+1	compile	26	trunk passed
+1	checkstyle	20	trunk passed
+1	mvnsite	27	trunk passed
-1	shadedclient	87	branch has errors when building and testing our client artifacts.
+1	javadoc	17	trunk passed
0	spotbugs	42	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	39	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	27	the patch passed
+1	compile	25	the patch passed
+1	javac	25	the patch passed
+1	checkstyle	19	hadoop-tools/hadoop-distcp: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
+1	mvnsite	27	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
-1	shadedclient	35	patch has errors when building and testing our client artifacts.
+1	javadoc	18	the patch passed
+1	findbugs	48	the patch passed
		_ Other Tests _
+1	unit	748	hadoop-distcp in the patch passed.
+1	asflicense	25	The patch does not generate ASF License warnings.
		1363

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/1/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ff9f9d667b0a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `36757ad`
Default Java	1.8.0_212
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/1/artifact/out/branch-mvninstall-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/1/testReport/
Max. process+thread count	392 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-07T02:57:36Z

hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyCommitter.java

+          null, taskAttemptContext);
+      try {
+        committer.commitJob(jobContext);
+


whitespace:end of line

hadoop-yetus · 2019-06-07T02:57:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	31	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1048	trunk passed
+1	compile	26	trunk passed
+1	checkstyle	19	trunk passed
+1	mvnsite	28	trunk passed
+1	shadedclient	657	branch has no errors when building and testing our client artifacts.
+1	javadoc	23	trunk passed
0	spotbugs	47	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	45	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	28	the patch passed
+1	compile	29	the patch passed
+1	javac	29	the patch passed
+1	checkstyle	18	hadoop-tools/hadoop-distcp: The patch generated 0 new + 218 unchanged - 2 fixed = 218 total (was 220)
+1	mvnsite	28	the patch passed
-1	whitespace	0	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	shadedclient	691	patch has no errors when building and testing our client artifacts.
+1	javadoc	16	the patch passed
+1	findbugs	45	the patch passed
		_ Other Tests _
+1	unit	759	hadoop-distcp in the patch passed.
+1	asflicense	26	The patch does not generate ASF License warnings.
		3609

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/2/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3efcdce2ec86 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3ea4f41`
Default Java	1.8.0_212
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/2/artifact/out/whitespace-eol.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/2/testReport/
Max. process+thread count	445 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-07T04:19:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	36	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1041	trunk passed
+1	compile	30	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	33	trunk passed
+1	shadedclient	743	branch has no errors when building and testing our client artifacts.
+1	javadoc	22	trunk passed
0	spotbugs	45	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	44	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	29	the patch passed
+1	compile	25	the patch passed
+1	javac	25	the patch passed
+1	checkstyle	19	hadoop-tools/hadoop-distcp: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
+1	mvnsite	27	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	738	patch has no errors when building and testing our client artifacts.
+1	javadoc	20	the patch passed
+1	findbugs	45	the patch passed
		_ Other Tests _
+1	unit	784	hadoop-distcp in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3776

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/3/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e8b6f06d366f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `a91d24f`
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/3/testReport/
Max. process+thread count	445 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-07T05:44:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	26	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1147	trunk passed
+1	compile	28	trunk passed
+1	checkstyle	21	trunk passed
+1	mvnsite	30	trunk passed
+1	shadedclient	765	branch has no errors when building and testing our client artifacts.
+1	javadoc	21	trunk passed
0	spotbugs	43	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	41	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	24	the patch passed
+1	compile	22	the patch passed
+1	javac	22	the patch passed
+1	checkstyle	16	hadoop-tools/hadoop-distcp: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
+1	mvnsite	26	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	814	patch has no errors when building and testing our client artifacts.
+1	javadoc	18	the patch passed
+1	findbugs	46	the patch passed
		_ Other Tests _
+1	unit	744	hadoop-distcp in the patch passed.
+1	asflicense	27	The patch does not generate ASF License warnings.
		3905

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/4/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 927dcfd4b098 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `a91d24f`
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/4/testReport/
Max. process+thread count	319 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-19T13:41:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	38	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1086	trunk passed
+1	compile	26	trunk passed
+1	checkstyle	18	trunk passed
+1	mvnsite	29	trunk passed
+1	shadedclient	656	branch has no errors when building and testing our client artifacts.
+1	javadoc	22	trunk passed
0	spotbugs	38	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	37	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	25	the patch passed
+1	compile	22	the patch passed
+1	javac	22	the patch passed
+1	checkstyle	15	hadoop-tools/hadoop-distcp: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
+1	mvnsite	27	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	690	patch has no errors when building and testing our client artifacts.
+1	javadoc	18	the patch passed
+1	findbugs	46	the patch passed
		_ Other Tests _
+1	unit	751	hadoop-distcp in the patch passed.
+1	asflicense	27	The patch does not generate ASF License warnings.
		3617

Subsystem	Report/Notes
Docker	Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/5/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 06d8e5a0ba79 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6282c02`
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/5/testReport/
Max. process+thread count	412 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-26T13:50:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	14	#919 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#919
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/6/console
versions	git=2.7.4
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-02T00:57:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	12	#919 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#919
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/7/console
versions	git=2.7.4
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java

hadoop-yetus · 2019-08-08T02:57:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	11	#919 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#919
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/8/console
versions	git=2.7.4
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

jojochuang · 2019-08-08T03:29:34Z

@kai33 could you check if a rebase is needed?

kai33 · 2019-08-08T06:08:13Z

I'll take a look

…

On Thu, Aug 8, 2019, 11:30 AM Wei-Chiu Chuang ***@***.***> wrote: @kai33 <https://github.com/kai33> could you check if a rebase is needed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#919?email_source=notifications&email_token=AA3MZOYFDM5TDBWCYCCKKDLQDOHN7A5CNFSM4HVDYYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32KCUY#issuecomment-519348563>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3MZO2LT5744TOZSV44HIDQDOHN7ANCNFSM4HVDYYSA> .

hadoop-yetus · 2019-08-08T11:03:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	13	#919 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#919
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/9/console
versions	git=2.7.4
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

jojochuang · 2019-08-12T18:59:59Z

Thanks. Retriggering the precommit to verify

hadoop-yetus · 2019-08-12T20:02:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1137	trunk passed
+1	compile	26	trunk passed
+1	checkstyle	19	trunk passed
+1	mvnsite	28	trunk passed
+1	shadedclient	666	branch has no errors when building and testing our client artifacts.
+1	javadoc	22	trunk passed
0	spotbugs	39	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	36	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	25	the patch passed
+1	compile	19	the patch passed
+1	javac	19	the patch passed
+1	checkstyle	19	hadoop-tools/hadoop-distcp: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
+1	mvnsite	23	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	716	patch has no errors when building and testing our client artifacts.
+1	javadoc	17	the patch passed
+1	findbugs	43	the patch passed
		_ Other Tests _
+1	unit	783	hadoop-distcp in the patch passed.
+1	asflicense	32	The patch does not generate ASF License warnings.
		3760

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/10/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e897f577ee28 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e4b538b`
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/10/testReport/
Max. process+thread count	422 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/10/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java

hadoop-yetus · 2019-08-16T17:56:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1059	trunk passed
+1	compile	20	trunk passed
+1	checkstyle	18	trunk passed
+1	mvnsite	27	trunk passed
+1	shadedclient	676	branch has no errors when building and testing our client artifacts.
+1	javadoc	19	trunk passed
0	spotbugs	37	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	34	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	24	the patch passed
+1	compile	22	the patch passed
+1	javac	22	the patch passed
+1	checkstyle	16	hadoop-tools/hadoop-distcp: The patch generated 0 new + 220 unchanged - 2 fixed = 220 total (was 222)
+1	mvnsite	22	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	759	patch has no errors when building and testing our client artifacts.
+1	javadoc	15	the patch passed
+1	findbugs	43	the patch passed
		_ Other Tests _
+1	unit	845	hadoop-distcp in the patch passed.
+1	asflicense	35	The patch does not generate ASF License warnings.
		3757

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-919/11/artifact/out/Dockerfile
GITHUB PR	#919
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0922c842552d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e356e4f`
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/11/testReport/
Max. process+thread count	430 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-919/11/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

jojochuang

+1

…in parallel (#919) * DistCp to support checksum validation when copy blocks in parallel * address review comments * add checksums comparison test for combine mode (cherry picked from commit c765584) (cherry picked from commit b3c14d4) Conflicts: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java

…in parallel (#919) * DistCp to support checksum validation when copy blocks in parallel * address review comments * add checksums comparison test for combine mode (cherry picked from commit c765584)

…opy blocks in parallel (apache#919) * DistCp to support checksum validation when copy blocks in parallel * address review comments * add checksums comparison test for combine mode (cherry picked from commit c765584) (cherry picked from commit b3c14d4) Conflicts: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java (cherry picked from commit c1a2b29) Conflicts: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java Change-Id: I5cb8890af0744a89edafd034d8db0a5834308b38

…in parallel (apache#919) * DistCp to support checksum validation when copy blocks in parallel * address review comments * add checksums comparison test for combine mode

…in parallel (apache#919) * DistCp to support checksum validation when copy blocks in parallel * address review comments * add checksums comparison test for combine mode (cherry picked from commit c765584) (cherry picked from commit b3c14d4) Conflicts: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java

hadoop-yetus reviewed Jun 7, 2019

View reviewed changes

kai33 force-pushed the HADOOP-16158-trunk branch from 8220444 to 02ad72a Compare June 7, 2019 04:38

jojochuang reviewed Aug 6, 2019

View reviewed changes

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java Outdated Show resolved Hide resolved

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java Outdated Show resolved Hide resolved

DistCp to support checksum validation when copy blocks in parallel

f26004c

kai33 force-pushed the HADOOP-16158-trunk branch from 02ad72a to f26004c Compare August 12, 2019 15:58

address review comments

7b24010

jojochuang requested changes Aug 16, 2019

View reviewed changes

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java Show resolved Hide resolved

add checksums comparison test for combine mode

2dcb47d

jojochuang approved these changes Aug 18, 2019

View reviewed changes

jojochuang merged commit c765584 into apache:trunk Aug 19, 2019

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019

Make config immutable again (apache#919)

7f6da82

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel #919

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel #919

kai33 commented Jun 6, 2019

hadoop-yetus commented Jun 6, 2019

hadoop-yetus Jun 7, 2019

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jul 19, 2019

hadoop-yetus commented Jul 26, 2019

hadoop-yetus commented Aug 2, 2019

hadoop-yetus commented Aug 8, 2019

jojochuang commented Aug 8, 2019

kai33 commented Aug 8, 2019 via email

hadoop-yetus commented Aug 8, 2019

jojochuang commented Aug 12, 2019

hadoop-yetus commented Aug 12, 2019

hadoop-yetus commented Aug 16, 2019

jojochuang left a comment

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel #919

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel #919

Conversation

kai33 commented Jun 6, 2019

hadoop-yetus commented Jun 6, 2019

hadoop-yetus Jun 7, 2019

Choose a reason for hiding this comment

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jun 7, 2019

hadoop-yetus commented Jul 19, 2019

hadoop-yetus commented Jul 26, 2019

hadoop-yetus commented Aug 2, 2019

hadoop-yetus commented Aug 8, 2019

jojochuang commented Aug 8, 2019

kai33 commented Aug 8, 2019 via email

hadoop-yetus commented Aug 8, 2019

jojochuang commented Aug 12, 2019

hadoop-yetus commented Aug 12, 2019

hadoop-yetus commented Aug 16, 2019

jojochuang left a comment

Choose a reason for hiding this comment