Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1113. Remove default dependencies from hadoop-ozone project #490

Closed
wants to merge 2 commits into from

Conversation

elek
Copy link
Member

@elek elek commented Feb 15, 2019

There are two ways to define common dependencies with maven:

1.) put all the dependencies to the parent project and inherit them
2.) get all the dependencies via transitive dependencies

TLDR; I would like to switch from 1 to 2 in hadoop-ozone

My main problem with the first approach that all the child project get a lot of dependencies independent if they need them or not. Let's imagine that I would like to create a new project (for example a java csi implementation) It doesn't need ozone-client, ozone-common etc, in fact it conflicts with ozone-client. But these jars are always added as of now.

Using transitive dependencies is more safe: we can add the dependencies where we need them and all of the other dependent projects will use them.

See: https://issues.apache.org/jira/browse/HDDS-1113

@anuengineer
Copy link
Contributor

I thought we fixed this, but I get this error when I build with this patch applied.

[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-ozone-common ---
[WARNING]
Dependency convergence error for com.squareup.okio:okio:1.6.0 paths to dependency are:
+-org.apache.hadoop:hadoop-ozone-common:0.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdfs-client:3.2.0
+-com.squareup.okhttp:okhttp:2.7.5
+-com.squareup.okio:okio:1.6.0
and
+-org.apache.hadoop:hadoop-ozone-common:0.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdds-common:0.4.0-SNAPSHOT
+-io.jaegertracing:jaeger-client:0.33.1
+-io.jaegertracing:jaeger-thrift:0.33.1
+-com.squareup.okhttp3:okhttp:3.9.0
+-com.squareup.okio:okio:1.13.0

@anuengineer
Copy link
Contributor

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 26 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 15 Maven dependency ordering for branch
+1 mvninstall 1019 trunk passed
-1 compile 18 hadoop-ozone in trunk failed.
-1 mvnsite 18 hadoop-ozone in trunk failed.
-1 mvnsite 19 client in trunk failed.
-1 mvnsite 17 common in trunk failed.
-1 mvnsite 22 integration-test in trunk failed.
-1 mvnsite 21 objectstore-service in trunk failed.
-1 mvnsite 19 ozone-manager in trunk failed.
-1 mvnsite 42 s3gateway in trunk failed.
-1 mvnsite 18 tools in trunk failed.
+1 shadedclient 1860 branch has no errors when building and testing our client artifacts.
-1 javadoc 13 hadoop-ozone in trunk failed.
-1 javadoc 15 client in trunk failed.
-1 javadoc 17 common in trunk failed.
-1 javadoc 15 integration-test in trunk failed.
-1 javadoc 18 objectstore-service in trunk failed.
-1 javadoc 17 ozone-manager in trunk failed.
-1 javadoc 17 s3gateway in trunk failed.
-1 javadoc 14 tools in trunk failed.
_ Patch Compile Tests _
0 mvndep 8 Maven dependency ordering for patch
-1 mvninstall 14 hadoop-ozone in the patch failed.
-1 mvninstall 10 client in the patch failed.
-1 mvninstall 10 common in the patch failed.
-1 mvninstall 9 integration-test in the patch failed.
-1 mvninstall 10 objectstore-service in the patch failed.
-1 mvninstall 10 ozone-manager in the patch failed.
-1 mvninstall 10 s3gateway in the patch failed.
-1 mvninstall 11 tools in the patch failed.
-1 compile 11 hadoop-ozone in the patch failed.
-1 javac 11 hadoop-ozone in the patch failed.
-1 mvnsite 16 hadoop-ozone in the patch failed.
-1 mvnsite 10 client in the patch failed.
-1 mvnsite 10 common in the patch failed.
-1 mvnsite 10 integration-test in the patch failed.
-1 mvnsite 11 objectstore-service in the patch failed.
-1 mvnsite 12 ozone-manager in the patch failed.
-1 mvnsite 12 s3gateway in the patch failed.
-1 mvnsite 11 tools in the patch failed.
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 11 The patch has no ill-formed XML file.
+1 shadedclient 726 patch has no errors when building and testing our client artifacts.
-1 javadoc 17 hadoop-ozone in the patch failed.
-1 javadoc 14 client in the patch failed.
-1 javadoc 16 common in the patch failed.
-1 javadoc 15 integration-test in the patch failed.
-1 javadoc 15 objectstore-service in the patch failed.
-1 javadoc 15 ozone-manager in the patch failed.
-1 javadoc 16 s3gateway in the patch failed.
-1 javadoc 16 tools in the patch failed.
_ Other Tests _
-1 unit 17 hadoop-ozone in the patch failed.
-1 unit 14 client in the patch failed.
-1 unit 15 common in the patch failed.
-1 unit 15 integration-test in the patch failed.
-1 unit 15 objectstore-service in the patch failed.
-1 unit 15 ozone-manager in the patch failed.
-1 unit 15 s3gateway in the patch failed.
-1 unit 14 tools in the patch failed.
+1 asflicense 28 The patch does not generate ASF License warnings.
3334
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/Dockerfile
GITHUB PR #490
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux e851ef1fcbf6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 6cec906
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-compile-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_client.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_common.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_objectstore-service.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_s3gateway.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-mvnsite-hadoop-ozone_tools.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_client.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_common.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_integration-test.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_objectstore-service.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_ozone-manager.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_s3gateway.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/branch-javadoc-hadoop-ozone_tools.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_client.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_common.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_objectstore-service.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_ozone-manager.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_s3gateway.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvninstall-hadoop-ozone_tools.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-compile-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_client.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_common.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_objectstore-service.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_ozone-manager.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_s3gateway.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-mvnsite-hadoop-ozone_tools.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_client.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_common.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_integration-test.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_objectstore-service.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_ozone-manager.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_s3gateway.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-javadoc-hadoop-ozone_tools.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_client.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_objectstore-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_ozone-manager.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_s3gateway.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/artifact/out/patch-unit-hadoop-ozone_tools.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/testReport/
Max. process+thread count 410 (vs. ulimit of 5500)
modules C: hadoop-ozone hadoop-ozone/client hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/objectstore-service hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/tools U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-490/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 23 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 77 Maven dependency ordering for branch
+1 mvninstall 977 trunk passed
-1 compile 93 hadoop-ozone in trunk failed.
-1 mvnsite 98 hadoop-ozone in trunk failed.
+1 shadedclient 2133 branch has no errors when building and testing our client artifacts.
+1 javadoc 221 trunk passed
_ Patch Compile Tests _
0 mvndep 13 Maven dependency ordering for patch
+1 mvninstall 501 the patch passed
-1 compile 97 hadoop-ozone in the patch failed.
-1 javac 97 hadoop-ozone in the patch failed.
-1 mvnsite 91 hadoop-ozone in the patch failed.
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 11 The patch has no ill-formed XML file.
+1 shadedclient 760 patch has no errors when building and testing our client artifacts.
+1 javadoc 203 the patch passed
_ Other Tests _
-1 unit 608 hadoop-ozone in the patch failed.
+1 unit 38 common in the patch passed.
+1 unit 28 client in the patch passed.
+1 unit 37 ozone-manager in the patch passed.
+1 unit 28 objectstore-service in the patch passed.
+1 unit 37 s3gateway in the patch passed.
-1 unit 539 integration-test in the patch failed.
-1 unit 74 tools in the patch failed.
+1 asflicense 31 The patch does not generate ASF License warnings.
5788
Reason Tests
Failed junit tests hadoop.ozone.ozShell.TestOzoneShell
hadoop.ozone.freon.TestFreonWithDatanodeFastRestart
hadoop.ozone.freon.TestRandomKeyGenerator
hadoop.ozone.ozShell.TestOzoneShell
hadoop.ozone.freon.TestFreonWithDatanodeFastRestart
hadoop.ozone.freon.TestRandomKeyGenerator
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/Dockerfile
GITHUB PR #490
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux 5b565315a3f0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 6c8c422
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/branch-compile-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/branch-mvnsite-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-compile-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-mvnsite-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-unit-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/artifact/out/patch-unit-hadoop-ozone_tools.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/testReport/
Max. process+thread count 3725 (vs. ulimit of 5500)
modules C: hadoop-ozone hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/s3gateway hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-490/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@arp7 arp7 self-requested a review March 5, 2019 18:31
Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 lgtm.

@elek
Copy link
Member Author

elek commented Mar 6, 2019

Thanks @arp7 the review, I am merging it to the trunk, right now.

@elek elek closed this in 45f976f Mar 6, 2019
elek added a commit that referenced this pull request Mar 7, 2019
…tributed by Elek, Marton.

Closes #490

(cherry picked from commit 45f976f)
shanthoosh added a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…ersionUpgrade.

Prevent invalid state updations on barrier.
* Introduced a additional barrier state NEW.
* Add state validations before updating the barrier.
* Fix existing TestZkBarrier tests that are disabled and add new tests
  to verify the intended behavior.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>
Author: Shanthoosh Venkataraman <santhoshvenkat1988@gmail.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes apache#490 from shanthoosh/fix_barrier_state_transitions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants