KAFKA-8634: Update ZooKeeper to 3.5.5 #6802

ijuma · 2019-05-23T22:00:04Z

ZooKeeper 3.5.5 is the first stable release in the 3.5.x series. The key new feature
in is TLS support, but there are a few more noteworthy features:

Dynamic reconfiguration
Local sessions
New node types: Container, TTL
Ability to remove watchers
Multi-threaded commit processor
Upgraded to Netty 4.1

See the release notes for more detail:
https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html

In addition to the version bump, we:

Add commons-cli dependency as it's required by ZooKeeperMain, but specified as
provided in their pom.
Remove unnecessary ZooKeeperMainWrapper, the bug it worked around was fixed
upstream a long time ago.
Ignore non zero exit in one system test invocation of ZooKeeperMain.
ZooKeeperMainWrapper always returned 0 and ZooKeeperService.query relies
on that for correct behavior.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ijuma · 2019-07-03T17:29:54Z

Fixed an issue that was causing system tests to fail and launched another build, which is looking good so far:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2772/console

ijuma · 2019-07-05T14:52:22Z

There were 10 system tests failures. Fixed an additional issue and started another run:

https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2779/

…but actually needed for the system tests to pass

ijuma · 2019-07-05T23:44:43Z

2 variations of ConnectDistributedTest failed in the latest run. Will investigate if it's related.

ijuma · 2019-07-06T01:54:21Z

retest this please

stanislavkozlovski · 2019-07-07T22:18:31Z

The org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector test failed on both JDK 11 builds with the error of

org.apache.kafka.connect.errors.DataException: Insufficient records committed by connector simple-conn in 15000 millis. Records expected=2000, actual=1112
	at org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
	at org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)

This isn't the initial connect test that failed, right?

For completeness, the JDK 8 build failed the kafka.admin.LeaderElectionCommandTest.testTopicPartition test with

kafka.common.AdminCommandFailedException: Timeout waiting for election results
	at kafka.admin.LeaderElectionCommand$.electLeaders(LeaderElectionCommand.scala:134)
	at kafka.admin.LeaderElectionCommand$.run(LeaderElectionCommand.scala:89)
	at kafka.admin.LeaderElectionCommand$.main(LeaderElectionCommand.scala:42)
	at kafka.admin.LeaderElectionCommandTest$$anonfun$testTopicPartition$1.apply(LeaderElectionCommandTest.scala:121)
	at kafka.admin.LeaderElectionCommandTest$$anonfun$testTopicPartition$1.apply(LeaderElectionCommandTest.scala:104)
	at kafka.utils.TestUtils$.resource(TestUtils.scala:1534)
	at kafka.admin.LeaderElectionCommandTest.testTopicPartition(LeaderElectionCommandTest.scala:104)

which seems more like a flake

ijuma · 2019-07-08T03:05:46Z

@stanislavkozlovski the JUnit failures are all flakes. The system test failures are not yet clear.

ijuma · 2019-07-08T08:27:07Z

The Connect system test that failed also looks like a flake, trunk also fails sometimes:

http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2019-07-05--001.1562343232--confluentinc--master--8dc8e6d/report.html

ijuma · 2019-07-08T08:27:14Z

retest this please

ijuma · 2019-07-08T16:48:24Z

I reran the connect system tests a few times and the failures are flaky (i.e. they pass sometimes). So, I think this PR is ready for review.

hachikuji

LGTM

Apache Kafka has upgraded to ZooKeeper 3.5.5 recently: apache/kafka#6802 Kafka has also removed `ZkUtils` and the `zkclient` dependency. ZooKeeper 3.5.5 is the first stable release in the 3.5.x series. The key new feature in is TLS support, but there are a few more noteworthy features: * Dynamic reconfiguration * Local sessions * New node types: Container, TTL * Ability to remove watchers * Multi-threaded commit processor * Upgraded to Netty 4.1 See the release notes for more detail: https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html

Apache Kafka has upgraded to ZooKeeper 3.5.5 recently: apache/kafka#6802 Kafka has also removed `ZkUtils` and the `zkclient` dependency. We do the same here (remove the `ZkUtils` clone and the `zkclient` dependency). We keep the `dependencyManagement` entries for `zookeeper` and `zkclient` as other Confluent projects have dependencies on those libraries. ZooKeeper 3.5.5 is the first stable release in the 3.5.x series. The key new feature is TLS support, but there are a few more noteworthy features: * Dynamic reconfiguration * Local sessions * New node types: Container, TTL * Ability to remove watchers * Multi-threaded commit processor * Upgraded to Netty 4.1 See the release notes for more detail: https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html

ijuma force-pushed the zookeeper-3.5 branch from 058dc4f to d29a4ce Compare July 3, 2019 05:29

ijuma marked this pull request as ready for review July 3, 2019 17:30

ijuma changed the title ~~[WIP] MINOR: Update ZooKeeper to 3.5.5~~ MINOR: Update ZooKeeper to 3.5.5 Jul 3, 2019

ijuma force-pushed the zookeeper-3.5 branch from e2f4de9 to b68b71d Compare July 5, 2019 04:42

ijuma added 5 commits July 5, 2019 07:53

MINOR: Update ZooKeeper to 3.5.5

48b6a9d

Add commons-cli dependency as that is a provided dependency by ZK, …

dcf2882

…but actually needed for the system tests to pass

Remove unnecessary ZooKeeperMainWrapper

2f92c55

Fix typo

f5bc668

Ignore non zero exit in the same way as the removed wrapper

e276428

ijuma force-pushed the zookeeper-3.5 branch from b68b71d to e276428 Compare July 5, 2019 14:54

ijuma requested a review from junrao July 8, 2019 08:27

ijuma changed the title ~~MINOR: Update ZooKeeper to 3.5.5~~ KAFKA-8634: Update ZooKeeper to 3.5.5 Jul 8, 2019

ijuma requested review from hachikuji and mumrah July 8, 2019 16:48

hachikuji approved these changes Jul 10, 2019

View reviewed changes

ijuma merged commit d67495d into apache:trunk Jul 10, 2019

ijuma deleted the zookeeper-3.5 branch July 10, 2019 16:45

ijuma mentioned this pull request Jul 11, 2019

Upgrade to ZooKeeper 3.5.5 and remove zkclient dependency confluentinc/common#211

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-8634: Update ZooKeeper to 3.5.5 #6802

KAFKA-8634: Update ZooKeeper to 3.5.5 #6802

ijuma commented May 23, 2019 •

edited

ijuma commented Jul 3, 2019

ijuma commented Jul 5, 2019

ijuma commented Jul 5, 2019

ijuma commented Jul 6, 2019

stanislavkozlovski commented Jul 7, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

hachikuji left a comment

KAFKA-8634: Update ZooKeeper to 3.5.5 #6802

KAFKA-8634: Update ZooKeeper to 3.5.5 #6802

Conversation

ijuma commented May 23, 2019 • edited

Committer Checklist (excluded from commit message)

ijuma commented Jul 3, 2019

ijuma commented Jul 5, 2019

ijuma commented Jul 5, 2019

ijuma commented Jul 6, 2019

stanislavkozlovski commented Jul 7, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

ijuma commented Jul 8, 2019

hachikuji left a comment

Choose a reason for hiding this comment

ijuma commented May 23, 2019 •

edited