New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka 4507: The client should send older versions of requests to the broker if necessary #2264
Conversation
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Previously, there were many places in the code that were creating AbstractRequest objects directly and passing them to the NetworkClient. Unfortunately, AbstractRequest objects are version-specific. For example, a ListOffsetRequest object of version 0 will have different fields inside its Struct than a ListOffsetRequest object of version 1. As soon as the message constructor is invoked, the contained Structs are formatted in the version-specific format. This change adds the AbstractRequest#Builder types. Client code in the Fetcher and Producer, and other places, can create these objects and pass them to the NetworkClient. The NetworkClient can then decide what version of each request to use based on the ApiVersionsRequest data. When the version that needs to be used is too old to support a necessary feature, an error is returned to the upper layer, which can then decide what to do. |
@cmccabe, can you please rebase this branch? Thanks! |
Rebased on trunk and fixed some unit tests |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Sorry, can you please rebase again? |
Rebased |
Also removed a few places where this change duplicated KAFKA-4548, and reverted a change to the test log4j.properties file |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Fix AuthorizerIntegrationTest, FetchRequestTest, SaslApiVersionsRequestTest, PlaintextConsumerTest. Fixed a bug where Fetcher#beginningOffsets and Fetcher#endOffsets incorrectly required a broker with the new ListOffsetsRequest RPC. Will add this to CompatibilityTest. |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
* make nodeApiVersions keyed on String, not Integer * some logging and whitespace cleanups * NodeApiVersions#getUsableVersion takes an ApiKeys object, not a short
…dateMetadataRequest version to 0
* Better encapsulation in `Protocol` * Higher-level code in NodeApiVersions * Remove unneeded casts * And more
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes, LGTM. We can address remaining issues in follow-ups. @cmccabe, please file a JIRA and link it from here when you can.
For the record, we had a known transient system test failure (ZooKeeperSecurityUpgradeTest.test_zk_security_upgrade
), but it passed on rerun.
I fixed some generic unchecked warnings before merging to trunk.
Refer to this link for build results (access rights to CI server needed): |
The client should send older versions of requests to the broker if necessary. Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2264 from cmccabe/KAFKA-4507
KAFKA-4507
The client should send older versions of requests to the broker if necessary.
Note: This builds on top of KAFKA-3600, which has not yet been committed yet.