Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upKafka: Add broker-level metrics-collecting filter #8188
Conversation
This comment has been minimized.
This comment has been minimized.
repokitteh
bot
commented
Sep 10, 2019
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
…eption handling in broker-level filter Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
… tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
…sts and use these utils instead Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
This comment has been minimized.
This comment has been minimized.
|
@mattklein123 I have changed the test to be Given it's an integration test and it might run differently depending on machine speed, virtualization, other factors I can't predict today => WDYT about making it manual for now, and me working on making it become part of normal (automated) build in future. Now waiting for the build to go green. /wait |
This comment has been minimized.
This comment has been minimized.
Sure sounds good. Very excited for this to land and to figure out what we can build on top. |
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
This comment has been minimized.
This comment has been minimized.
|
/wait (will re-request review when I finally get it to green) |
…me during int tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
This comment has been minimized.
This comment has been minimized.
|
/wait-any |
This comment has been minimized.
This comment has been minimized.
|
Ready for re-review. |
|
Absolutely epic work. Let's ship an iterate! |
a60f685
into
envoyproxy:master
| sha256 = "ae7a1696c0a0302b43c5b21e515c37e6ecd365941f68a510a7e442eebddf39a1", # 2.2.0-rc2 | ||
| strip_prefix = "kafka-2.2.0-rc2/clients/src/main/resources/common/message", | ||
| urls = ["https://github.com/apache/kafka/archive/2.2.0-rc2.zip"], |
This comment has been minimized.
This comment has been minimized.
moderation
Jan 6, 2020
Contributor
Congrats on shipping this @adamkotwasinski. I suspect this version lines up with when you started the PR. Kafka is now at 2.4.0 - https://github.com/apache/kafka/releases/tag/2.4.0. Are there any blockers on moving from the old RC release to the latest stable release?
This comment has been minimized.
This comment has been minimized.
adamkotwasinski
Jan 6, 2020
Author
Contributor
@moderation there should be none - I'll take a look as soon as I can
This comment has been minimized.
This comment has been minimized.
adamkotwasinski
Jan 7, 2020
Author
Contributor
@moderation work will be in #9582 ; I will need some more time to get the 2.4 running (the descriptor files used to generate the C++ code changed a little)
| strip_prefix = "kafka_2.12-2.2.0", | ||
| urls = ["http://us.mirrors.quenda.co/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz"], |
This comment has been minimized.
This comment has been minimized.
moderation
Jan 6, 2020
Contributor
Same for the server binary - http://us.mirrors.quenda.co/apache/kafka/2.4.0/
adamkotwasinski commentedSep 9, 2019
•
edited
Description: Simple Kafka filter for broker. Grabs and decodes messages and updates metrics.
If message could not be recognised (the protocol used by client<->broker comms is higher than what's present in Envoy), we just pass through the payloads (IMHO it would be bad to have Envoy impact communication in this particular filter - this is not going to be possible in "mesh filter" as we we'd need).
In detail the change includes:
messaging_utilitiestest code library that's capable of making example payloads of various kinds (is going to be used for further tests).Relates to #2852
Risk Level: Low (Kafka code is unused right now)
Testing: automated tests, manual testing with real Kafka broker and client
Docs Changes: Kafka broker filter added
Release Notes: n/a