-
Notifications
You must be signed in to change notification settings - Fork 780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for flexible messages #562
Conversation
working_directory: *working_directory | ||
environment: | ||
KAFKA_VERSION: "2.4.1" | ||
KAFKA_SKIP_NETTEST: "1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mind explaining why these tests cannot pass with 2.4?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Also added a similar comment to where KAFKA_SKIP_NETTEST
is read in the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking great 🙌
Description
This change updates kafka-go to support the "flexible" message format described in KIP-482. Among other changes, the new format adds optional "tagged fields" into API messages and also makes the encodings of strings and arrays more compact.
The values of the tagged fields aren't used for much yet (I think), but we need to support the new, "flexible" format to get the latest API versions described in the Kafka protocol guide.
The proposed interface is to support tagged fields via a new,
tag=
property in the existing protocol struct tags, e.g.:As an example and to provide something to test with, this change updates the max version of the
CreateTopics
API to v5, which uses the new format.Misc. notes and TODOs
nettest
suite fails for this version, so I'm temporarily adding an option to skip this. I get the same failures when running from the base0.4
branch, so I don't think these errors are related to the changes here.VarString
,VarBytes
, etc. to distinguish them.