Support Fetch v10 protocol (minimal) #792
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test failures are unrelated, see #786 |
ankon
force-pushed
the
pr/fetch-v10
branch
2 times, most recently
from
a50eaca
to
b9effb0
Compare
Nevon
reviewed
Jul 20, 2020
This version is identical to v9, but allows for the compression type to be 'zstd', and allows a new error code to be returned when fetching. We already had earlier code to inject a zstd codec through the regular codec extension mechanism, this commit now simply makes it possible to actually trigger that code.
|
So this PR is pretty simple and uncontroversial, but I want to make sure that I understand what the implications are. By supporting Fetch V10, the only thing that will happen is that if a partition is using zstd compression, we will get back data (that we can't decompress without a codec), rather than an |
|
Yes, this is my understanding as well. KafkaJS supports the v2 format from the KIP, so the discussion about down-conversion shouldn't apply at all. By sending a Fetch v10 request we're indicating that we understand a response that talks about zstd (which we do-ish, assuming we had a codec). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This provides the protocol definitions for Fetch v10 (KIP-110: ZStd Support), without any additional changes to actually support consuming or producing zstd compressed content.
For producing: This will require updating the produce APIs as well, see https://github.com/tulios/kafkajs/projects/2 for the bigger picture.
For consuming: This might actually work just fine, if one provides a suitable codec through the regular extension mechanism.
This is mainly here as a prerequisite for KIP-392 support (#713), which requires Fetch v11 messages.
Note: This PR is floating on top of the Fetch v9 changes (#778), and should stay a draft until we can merge that one first.