HDFS-14824. [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.

[tasanuma]

No description provided.

[tasanuma]

I almost just merged linkedin/dynamometer#76, and fixed the license and checkstyle warns.

I tested this PR with my test cluster and the result seems good.

$ hadoop fs -cat dyno/results/part-r-00000
userA,READ       171
userA,WRITE      2772
userB,READ       22
...

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	2141	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	39	Maven dependency ordering for branch
+1	mvninstall	1295	trunk passed
+1	compile	28	trunk passed
+1	checkstyle	21	trunk passed
+1	mvnsite	47	trunk passed
+1	shadedclient	889	branch has no errors when building and testing our client artifacts.
+1	javadoc	41	trunk passed
0	spotbugs	36	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	71	trunk passed
		_ Patch Compile Tests _
0	mvndep	12	Maven dependency ordering for patch
+1	mvninstall	46	the patch passed
+1	compile	24	the patch passed
+1	javac	24	the patch passed
+1	checkstyle	15	the patch passed
+1	mvnsite	37	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	880	patch has no errors when building and testing our client artifacts.
+1	javadoc	36	the patch passed
+1	findbugs	78	the patch passed
		_ Other Tests _
-1	unit	33	hadoop-dynamometer-workload in the patch failed.
-1	unit	22	hadoop-dynamometer-infra in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		5877

Reason	Tests
Failed junit tests	hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator
	hadoop.tools.dynamometer.TestDynamometerInfra

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/1/artifact/out/Dockerfile
GITHUB PR	#1685
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 29685438c85d 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 012756a
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/1/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-workload.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/1/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/1/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra U: hadoop-tools/hadoop-dynamometer
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[tasanuma]

Updated PR. I tested it again with my test cluster and the result is like below.

$ hadoop fs -cat dyno/results/part-r-00000
...
userA,READ,GETFILEINFO,462,6476
userA,READ,LISTSTATUS,441,8821
userA,READ,OPEN,168,2409
...

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	38	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	59	Maven dependency ordering for branch
+1	mvninstall	1114	trunk passed
+1	compile	35	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	96	trunk passed
+1	shadedclient	942	branch has no errors when building and testing our client artifacts.
+1	javadoc	84	trunk passed
0	spotbugs	57	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	134	trunk passed
		_ Patch Compile Tests _
0	mvndep	15	Maven dependency ordering for patch
+1	mvninstall	90	the patch passed
+1	compile	28	the patch passed
+1	javac	28	the patch passed
+1	checkstyle	17	the patch passed
+1	mvnsite	77	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	818	patch has no errors when building and testing our client artifacts.
+1	javadoc	72	the patch passed
+1	findbugs	142	the patch passed
		_ Other Tests _
+1	unit	55	hadoop-dynamometer-workload in the patch passed.
-1	unit	27	hadoop-dynamometer-infra in the patch failed.
-1	unit	82	hadoop-dynamometer in the patch failed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		4049

Reason	Tests
Failed junit tests	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.tools.dynamometer.TestDynamometerInfra

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/2/artifact/out/Dockerfile
GITHUB PR	#1685
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux d5323e292dc8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / e6137d0
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/2/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/2/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/2/testReport/
Max. process+thread count	418 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-dynamometer U: hadoop-tools/hadoop-dynamometer
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[tasanuma]

The failed test seems not to be related.

@xkrogen Could you review it?

[xkrogen]

Hi @tasanuma , the changes LGTM. The test is failing because 3.1.2 no longer exists on the Apache mirror sites now that 3.1.3 is published. Can you also take care of updating that version here to fix the test? We can use 3.1.3 or 3.2.1. The long term fix of removing this specification is tracked in HDFS-14412.

[tasanuma]

Thanks for your review, @xkrogen. Updated to 3.1.3.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1784	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	35	Maven dependency ordering for branch
+1	mvninstall	1099	trunk passed
+1	compile	35	trunk passed
+1	checkstyle	24	trunk passed
+1	mvnsite	96	trunk passed
+1	shadedclient	909	branch has no errors when building and testing our client artifacts.
+1	javadoc	82	trunk passed
0	spotbugs	55	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	129	trunk passed
		_ Patch Compile Tests _
0	mvndep	13	Maven dependency ordering for patch
+1	mvninstall	87	the patch passed
+1	compile	26	the patch passed
+1	javac	26	the patch passed
+1	checkstyle	16	the patch passed
+1	mvnsite	72	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	822	patch has no errors when building and testing our client artifacts.
+1	javadoc	67	the patch passed
+1	findbugs	146	the patch passed
		_ Other Tests _
+1	unit	52	hadoop-dynamometer-workload in the patch passed.
+1	unit	482	hadoop-dynamometer-infra in the patch passed.
+1	unit	533	hadoop-dynamometer in the patch passed.
+1	asflicense	31	The patch does not generate ASF License warnings.
		6608

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/3/artifact/out/Dockerfile
GITHUB PR	#1685
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux aed26f178d37 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / ef9d12d
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/3/testReport/
Max. process+thread count	989 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-dynamometer U: hadoop-tools/hadoop-dynamometer
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1685/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[xkrogen]

Thanks @tasanuma !