HDDS-2120. Remove hadoop classes from ozonefs-current jar

[elek]

We have two kind of ozone file system jars: current and legacy. current is designed to work only with exactly the same hadoop version which is used for compilation (3.2 as of now).

But as of now the hadoop classes are included in the current jar which is not necessary as the jar is expected to be used in an environment where the hadoop classes (exactly the same hadoop classes) are already there. They can be excluded.

See: https://issues.apache.org/jira/browse/HDDS-2120

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
-1	mvninstall	33	hadoop-ozone in trunk failed.
-1	compile	20	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	1043	branch has no errors when building and testing our client artifacts.
-1	javadoc	16	hadoop-hdds in trunk failed.
-1	javadoc	14	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	30	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	716	patch has no errors when building and testing our client artifacts.
-1	javadoc	13	hadoop-hdds in the patch failed.
-1	javadoc	14	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	134	hadoop-hdds in the patch failed.
-1	unit	25	hadoop-ozone in the patch failed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		2391

Reason	Tests
Failed junit tests	hadoop.ozone.container.ozoneimpl.TestOzoneContainer
	hadoop.ozone.container.keyvalue.TestKeyValueContainer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/Dockerfile
GITHUB PR	#1434
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 6db3fe47a36e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 1505d3f
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/testReport/
Max. process+thread count	304 (vs. ulimit of 5500)
modules	C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1434/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[arp7]

Thanks for the quick fix @elek. Should we also exclude Ratis.. since HBase potentially has a dependency on Ratis and we should use the in-build version instead.

What about third party dependencies? Downstream components could run into conflicts.. I believe that will require shading.

[elek]

Should we also exclude Ratis.. since HBase potentially has a dependency on Ratis and we should use the in-build version instead.

We need Ratis on the classpath. I am not sure what is the best approach for support HBase. If we exclude it, we need to put two jar files to the classpath of hive/spark: ozonefs and ratis.

Do you know which HBase version uses Ratis?

What about third party dependencies? Downstream components could run into conflicts.. I believe that will require shading

Yes, they can be shaded in HDDS-2121 (at least half of them. Some of them should be kept in original form such as protobuf or the logging).

[elek]

/retest

[avijayanhwx]

LGTM +1

[arp7]

+1

[arp7]

/retest

[elek]

Thanks the review @arp7 and @ajayydv

The test failures are not related. I am merging it to the trunk right now...

[arp7]

It looks like this failed compilation in Jenkins... does the Jenkins job need to be updated to use the separated pom compilation command?