[improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933

eolivelli · 2023-08-04T12:43:21Z

Motivation

Apache ZooKeeper 3.9.0 comes with a few performance improvements and also full support for backup/restore.
It is fully compatible with 3.8.x.
It is worth to upgrade for Pulsar 3.1.0

Release notes:
https://zookeeper.apache.org/doc/r3.9.0/releasenotes.html

Modifications

Upgrade Apache ZooKeeper
in update the base test class that starts ZK to use ZooKeeper Embedded

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: eolivelli#24

pulsar-metadata/src/test/java/org/apache/pulsar/metadata/TestZKServer.java

michaeljmarshall · 2023-08-09T14:20:10Z

pulsar-metadata/src/test/java/org/apache/pulsar/metadata/TestZKServer.java

 import org.assertj.core.util.Files;
+import org.junit.platform.commons.util.ReflectionUtils;


There is also an issue with this line. It is failing checkstyle because of this recent PR #20611

In looking into the ReflectionUtils source code, I see the following warning:

/** * Collection of utilities for working with the Java reflection APIs. * * <h2>DISCLAIMER</h2> * * <p>These utilities are intended solely for usage within the JUnit framework * itself. <strong>Any usage by external parties is not supported.</strong> * Use at your own risk! * * <p>Some utilities are published via the maintained {@code ReflectionSupport} * class. * * @since 1.0 * @see org.junit.platform.commons.support.ReflectionSupport */ @API(status = INTERNAL, since = "1.0") public final class ReflectionUtils

We might need to refactor this part of the PR. What is your opinion @eolivelli?

this is TestZKServer, it is testing code, so it is fine to use JUnit utilities IMHO

btw, I have removed it

github-actions · 2023-10-02T01:47:30Z

The pr had no activity for 30 days, mark with Stale label.

tisonkun

Also there are some license files should be updated.

lhotari · 2023-10-16T12:42:27Z

Please also upgrade to use 3.9.1

lhotari · 2023-10-18T10:24:04Z

@eolivelli Please rebase & upgrade to use 3.9.1 since that would be needed for solving CVE-2023-44981 .

eolivelli · 2023-10-18T12:30:40Z

@lhotari would you have time to pick up this PR please ? I don't have much time these days

lhotari · 2023-10-18T13:19:45Z

@lhotari would you have time to pick up this PR please ? I don't have much time these days

@eolivelli sure. I'll complete this PR.

…etty:netty-transport-native-epoll:jar` [INFO] +- org.apache.zookeeper:zookeeper-prometheus-metrics:jar:3.9.1:compile [INFO] | \- org.apache.zookeeper:zookeeper:jar:3.9.1:compile [INFO] | +- org.apache.zookeeper:zookeeper-jute:jar:3.9.1:compile [INFO] | \- org.apache.yetus:audience-annotations:jar:0.12.0:compile [INFO] +- org.apache.zookeeper:zookeeper:jar:tests:3.9.1:test [INFO] | +- org.apache.zookeeper:zookeeper-jute:jar:3.9.1:compile [INFO] | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile [INFO] | +- io.netty:netty-handler:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-resolver:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-transport:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-transport-native-unix-common:jar:4.1.100.Final:compile [INFO] | | \- io.netty:netty-codec:jar:4.1.100.Final:compile [INFO] | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.100.Final:compile [INFO] | | \- io.netty:netty-transport-classes-epoll:jar:4.1.100.Final:compile [INFO] | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.61.Final:compile [INFO] | | +- io.netty:netty-tcnative-classes:jar:2.0.61.Final:compile [INFO] | | +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.61.Final:compile [INFO] | | +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.61.Final:compile [INFO] | | +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.61.Final:compile [INFO] | | +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.61.Final:compile [INFO] | | \- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.61.Final:compile [INFO] | \- commons-io:commons-io:jar:2.8.0:compile [INFO] +- org.apache.zookeeper:zookeeper-prometheus-metrics:jar:3.8.1:compile [INFO] | \- org.apache.zookeeper:zookeeper:jar:3.8.1:compile [INFO] | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.1:compile [INFO] | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile [INFO] | \- io.netty:netty-transport-native-epoll:jar:4.1.100.Final:compile [INFO] +- org.apache.zookeeper:zookeeper:jar:tests:3.8.1:test [INFO] | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.1:compile [INFO] | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile [INFO] | +- io.netty:netty-handler:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-resolver:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-transport:jar:4.1.100.Final:compile [INFO] | | +- io.netty:netty-transport-native-unix-common:jar:4.1.100.Final:compile [INFO] | | \- io.netty:netty-codec:jar:4.1.100.Final:compile [INFO] | +- io.netty:netty-transport-native-epoll:jar:4.1.100.Final:test [INFO] | | \- io.netty:netty-transport-classes-epoll:jar:4.1.100.Final:compile [INFO] | \- commons-io:commons-io:jar:2.8.0:compile

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 5572c38)

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org>

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 5572c38)

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit a614aad) Conflicts: distribution/server/src/assemble/LICENSE.bin.txt pom.xml pulsar-sql/presto-distribution/LICENSE

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 5572c38)

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org>

zookeeper 3.9.1 add a check to print a log. ```java if (addr.getPort() == 0) { LOG.info("bound to port {}", this.ss.getLocalAddress()); } ```

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 5572c38)

lhotari · 2024-02-27T06:59:56Z

Cherry-picking to 2.11.4 for consistency. This is already cherry-picked to branch-2.10 .

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 5572c38) # Conflicts: # pom.xml

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Aug 4, 2023

eolivelli requested review from merlimat and tisonkun August 4, 2023 13:13

Technoboy- added this to the 3.2.0 milestone Aug 7, 2023

Technoboy- assigned eolivelli Aug 7, 2023

Technoboy- added ready-to-test dependencies Pull requests that update a dependency file labels Aug 7, 2023

Technoboy- closed this Aug 7, 2023

Technoboy- reopened this Aug 7, 2023

tisonkun changed the title ~~[broker][bookie] Upgrade Apache ZooKeeper to 3.9.0~~ [improve][build] Upgrade Apache ZooKeeper to 3.9.0 Aug 8, 2023

tisonkun reviewed Aug 8, 2023

View reviewed changes

pulsar-metadata/src/test/java/org/apache/pulsar/metadata/TestZKServer.java Show resolved Hide resolved

michaeljmarshall reviewed Aug 9, 2023

View reviewed changes

eolivelli force-pushed the impl/zookeeper-3.9.0 branch from 39f5b77 to cb18934 Compare September 1, 2023 07:54

github-actions bot added the Stale label Oct 2, 2023

tisonkun reviewed Oct 16, 2023

View reviewed changes

github-actions bot removed the Stale label Oct 17, 2023

lhotari changed the title ~~[improve][build] Upgrade Apache ZooKeeper to 3.9.0~~ [improve][build] Upgrade Apache ZooKeeper to 3.9.1 Oct 18, 2023

lhotari and others added 6 commits October 18, 2023 16:23

[broker][bookie] Upgrade Apache ZooKeeper to 3.9.0

b726bd7

Use ZooKeeper server Embedded

364d0dd

Fix tests

979e702

fix tests

9fc60ee

Fix style

4d72ef3

Fix import style

cc16964

lhotari force-pushed the impl/zookeeper-3.9.0 branch from a39d14c to cc16964 Compare October 18, 2023 13:29

lhotari requested a review from tisonkun October 18, 2023 13:29

lhotari requested a review from michaeljmarshall October 18, 2023 13:29

liangyepianzhou and others added 3 commits November 28, 2023 11:37

Merge remote-tracking branch 'apache/master' into impl/zookeeper-3.9.0

27b622a

add hamcrest dependency

dfa7bb1

Technoboy- approved these changes Nov 29, 2023

View reviewed changes

liangyepianzhou approved these changes Nov 29, 2023

View reviewed changes

Technoboy- merged commit 5572c38 into apache:master Nov 29, 2023
46 of 48 checks passed

Technoboy- added release/3.0.3 release/3.1.3 labels Nov 29, 2023

liangyepianzhou added the cherry-picked/branch-3.0 label Nov 29, 2023

Technoboy- added a commit that referenced this pull request Dec 4, 2023

[improve][build] Upgrade Apache ZooKeeper to 3.9.1 (#20933)

a614aad

Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: xiangying <1984997880@qq.com> Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- added the cherry-picked/branch-3.1 label Dec 4, 2023

liangyepianzhou added cherry-picked/branch-2.10 release/2.10.6 labels Dec 12, 2023

nikhil-ctds mentioned this pull request Dec 12, 2023

[fix][sec] Cherry-pick owasp-dependency-check fixes datastax/pulsar#211

Closed

4 tasks

lhotari added cherry-picked/branch-2.11 release/2.11.4 labels Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933

[improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933

eolivelli commented Aug 4, 2023 •

edited

Loading

michaeljmarshall Aug 9, 2023

michaeljmarshall Aug 9, 2023 •

edited

Loading

eolivelli Sep 1, 2023

eolivelli Sep 1, 2023

github-actions bot commented Oct 2, 2023

tisonkun left a comment

lhotari commented Oct 16, 2023

lhotari commented Oct 18, 2023

eolivelli commented Oct 18, 2023

lhotari commented Oct 18, 2023

lhotari commented Feb 27, 2024

		import org.assertj.core.util.Files;
		import org.junit.platform.commons.util.ReflectionUtils;

[improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933

[improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933

Conversation

eolivelli commented Aug 4, 2023 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

michaeljmarshall Aug 9, 2023

Choose a reason for hiding this comment

michaeljmarshall Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

eolivelli Sep 1, 2023

Choose a reason for hiding this comment

eolivelli Sep 1, 2023

Choose a reason for hiding this comment

github-actions bot commented Oct 2, 2023

tisonkun left a comment

Choose a reason for hiding this comment

lhotari commented Oct 16, 2023

lhotari commented Oct 18, 2023

eolivelli commented Oct 18, 2023

lhotari commented Oct 18, 2023

lhotari commented Feb 27, 2024

eolivelli commented Aug 4, 2023 •

edited

Loading

michaeljmarshall Aug 9, 2023 •

edited

Loading