HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.

[avijayanhwx]

With this patch, RocksDB logging can be configured for OM. If needed, we can write RocksDB logs to a separate file using a separate appender in log4j.properties.

[avijayanhwx]

/label ozone

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	74	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
-1	mvninstall	41	hadoop-hdds in trunk failed.
-1	mvninstall	31	hadoop-ozone in trunk failed.
-1	compile	18	hadoop-hdds in trunk failed.
-1	compile	14	hadoop-ozone in trunk failed.
+1	checkstyle	60	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	965	branch has no errors when building and testing our client artifacts.
-1	javadoc	21	hadoop-hdds in trunk failed.
-1	javadoc	17	hadoop-ozone in trunk failed.
0	spotbugs	1058	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	34	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	32	hadoop-hdds in the patch failed.
-1	mvninstall	26	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	52	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	793	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	29	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	25	hadoop-hdds in the patch failed.
-1	unit	21	hadoop-ozone in the patch failed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		2500

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/Dockerfile
GITHUB PR	#1538
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 9d91d864f742 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 9bf7a6e
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/testReport/
Max. process+thread count	339 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common U: hadoop-hdds/common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[avijayanhwx]

/retest

[anuengineer]

/retest

[anuengineer]

Can you please add a test case that proves that RocksDB actually produces logs that we can see. There is a log listener class in the Hadoop. You can use them, or create a test and then grep for some of the log statements. Otherwise the change looks quite good to me.

[avijayanhwx]

Can you please add a test case that proves that RocksDB actually produces logs that we can see. There is a log listener class in the Hadoop. You can use them, or create a test and then grep for some of the log statements. Otherwise the change looks quite good to me.

Thanks for the review @anuengineer . I have added unit test for verifying RocksDB logging.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	94	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
-1	mvninstall	40	hadoop-hdds in trunk failed.
-1	mvninstall	39	hadoop-ozone in trunk failed.
-1	compile	21	hadoop-hdds in trunk failed.
-1	compile	15	hadoop-ozone in trunk failed.
+1	checkstyle	65	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	934	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	17	hadoop-ozone in trunk failed.
0	spotbugs	1027	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	33	hadoop-hdds in trunk failed.
-1	findbugs	18	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
-1	mvninstall	34	hadoop-hdds in the patch failed.
-1	mvninstall	36	hadoop-ozone in the patch failed.
-1	compile	23	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	23	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	59	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	769	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	17	hadoop-ozone in the patch failed.
-1	findbugs	32	hadoop-hdds in the patch failed.
-1	findbugs	18	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	26	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2559

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/Dockerfile
GITHUB PR	#1538
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 9cb9436ccdd7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / a3fe404
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/testReport/
Max. process+thread count	305 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	99	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	73	Maven dependency ordering for branch
-1	mvninstall	39	hadoop-hdds in trunk failed.
-1	mvninstall	40	hadoop-ozone in trunk failed.
-1	compile	21	hadoop-hdds in trunk failed.
-1	compile	14	hadoop-ozone in trunk failed.
+1	checkstyle	65	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	957	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	19	hadoop-ozone in trunk failed.
0	spotbugs	1050	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	32	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	29	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	37	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	17	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	17	hadoop-ozone in the patch failed.
+1	checkstyle	57	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	782	patch has no errors when building and testing our client artifacts.
-1	javadoc	21	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	32	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	25	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	31	The patch does not generate ASF License warnings.
		2646

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/Dockerfile
GITHUB PR	#1538
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f018c43efef6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / a3fe404
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/testReport/
Max. process+thread count	305 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1538/2/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[anuengineer]

Thank you for the contribution. I have committed this to the trunk.