-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka: Add broker-level metrics-collecting filter #8188
Conversation
ce10704
to
8796af1
Compare
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
8796af1
to
8056d1e
Compare
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
af862dd
to
83d695d
Compare
…eption handling in broker-level filter Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
… tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
…sts and use these utils instead Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
@mattklein123 I have changed the test to be Given it's an integration test and it might run differently depending on machine speed, virtualization, other factors I can't predict today => WDYT about making it manual for now, and me working on making it become part of normal (automated) build in future. Now waiting for the build to go green. /wait |
Sure sounds good. Very excited for this to land and to figure out what we can build on top. |
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
/wait (will re-request review when I finally get it to green) |
…me during int tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
/wait-any |
Ready for re-review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Absolutely epic work. Let's ship an iterate!
sha256 = "ae7a1696c0a0302b43c5b21e515c37e6ecd365941f68a510a7e442eebddf39a1", # 2.2.0-rc2 | ||
strip_prefix = "kafka-2.2.0-rc2/clients/src/main/resources/common/message", | ||
urls = ["https://github.com/apache/kafka/archive/2.2.0-rc2.zip"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Congrats on shipping this @adamkotwasinski. I suspect this version lines up with when you started the PR. Kafka is now at 2.4.0 - https://github.com/apache/kafka/releases/tag/2.4.0. Are there any blockers on moving from the old RC release to the latest stable release?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@moderation there should be none - I'll take a look as soon as I can
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@moderation work will be in #9582 ; I will need some more time to get the 2.4 running (the descriptor files used to generate the C++ code changed a little)
strip_prefix = "kafka_2.12-2.2.0", | ||
urls = ["http://us.mirrors.quenda.co/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same for the server binary - http://us.mirrors.quenda.co/apache/kafka/2.4.0/
Description: Simple Kafka filter for broker. Grabs and decodes messages and updates metrics.
If message could not be recognised (the protocol used by client<->broker comms is higher than what's present in Envoy), we just pass through the payloads (IMHO it would be bad to have Envoy impact communication in this particular filter - this is not going to be possible in "mesh filter" as we we'd need).
In detail the change includes:
messaging_utilities
test code library that's capable of making example payloads of various kinds (is going to be used for further tests).Relates to #2852
Risk Level: Low (Kafka code is unused right now)
Testing: automated tests, manual testing with real Kafka broker and client
Docs Changes: Kafka broker filter added
Release Notes: n/a