KAFKA-13270: Set JUTE_MAXBUFFER to 4 MB by default #11295
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@omkreddy if you have cycles, this is a 3.0 blocker. |
dajac
reviewed
Sep 6, 2021
| getZooKeeperClientProperty(zkClientConfig, ZkClientCnxnSocketProp).isDefined && | ||
| getZooKeeperClientProperty(zkClientConfig, ZkSslKeyStoreLocationProp).isDefined | ||
| private[kafka] def zkTlsClientAuthEnabled(zkClientConfig: ZKClientConfig): Boolean = { | ||
| zooKeeperClientProperty(zkClientConfig, ZkSslClientEnableProp).map(_ == "true").getOrElse(false) && |
There was a problem hiding this comment.
nit: We could use exists here.
There was a problem hiding this comment.
True, even better: contains. :)
| val prefixedValue = configMap.get(AclAuthorizer.configPrefix + kafkaProp) | ||
| if (prefixedValue.isDefined) | ||
| zkClientConfig.get.setProperty(sysProp, | ||
| KafkaConfig.ZkSslConfigToSystemPropertyMap.forKeyValue { (kafkaProp, sysProp) => { |
There was a problem hiding this comment.
nit: We could remove the extra block defined by the last { on this line.
Comment on lines
1356
to
1360
| val client1 = KafkaZkClient(zkConnect, zkAclsEnabled.getOrElse(JaasUtils.isZkSaslEnabled), zkSessionTimeout, | ||
| zkConnectionTimeout, zkMaxInFlightRequests, Time.SYSTEM, name = "KafkaZkClient", | ||
| zkClientConfig = clientConfig1) | ||
| try assertEquals("2048000", client1.currentZooKeeper.getClientConfig.getProperty(ZKConfig.JUTE_MAXBUFFER)) | ||
| finally client1.close() |
There was a problem hiding this comment.
Is it worth defining an inner helper method to avoid the repeated code here? Something like: assertPropery(config: ZKClientConfig, property: String, expectedValue: String).
There was a problem hiding this comment.
That's a fair suggestion, I pushed a change along these lines.
|
@dajac Thanks for the review, I pushed a commit that addresses your feedback. |
ijuma
added a commit
that referenced
this pull request
Sep 6, 2021
We restore the 3.4.x/3.5.x behavior unless the caller has set the property (note that ZKConfig auto configures itself if certain system properties have been set). I added a unit test that fails without the change and passes with it. I also refactored the code to streamline the way we handle parameters passed to KafkaZkClient and ZooKeeperClient. See apache/zookeeper#1129 for the details on why the behavior changed in 3.6.0. Credit to @rondagostino for finding and reporting this issue. Reviewers: David Jacot <djacot@confluent.io>
xdgrulez
pushed a commit
to xdgrulez/kafka
that referenced
this pull request
Dec 22, 2021
We restore the 3.4.x/3.5.x behavior unless the caller has set the property (note that ZKConfig auto configures itself if certain system properties have been set). I added a unit test that fails without the change and passes with it. I also refactored the code to streamline the way we handle parameters passed to KafkaZkClient and ZooKeeperClient. See apache/zookeeper#1129 for the details on why the behavior changed in 3.6.0. Credit to @rondagostino for finding and reporting this issue. Reviewers: David Jacot <djacot@confluent.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ZooKeeper 3.6.0 changed the default configuration for JUTE_MAXBUFFER from 4 MB to 1 MB.
This causes a regression if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message of the form
"java.io.IOException: Packet len <####> is out of range".
We restore the 3.4.x/3.5.x behavior unless the caller has set the property (note that ZKConfig
auto configures itself if certain system properties have been set).
I added a unit test that fails without the change and passes with it.
I also refactored the code to streamline the way we handle parameters passed to
KafkaZkClient and ZooKeeperClient.
See apache/zookeeper#1129 for the details on why the behavior
changed in 3.6.0.
Credit to @rondagostino for finding and reporting this issue.
Committer Checklist (excluded from commit message)