Kafka: Add broker-level metrics-collecting filter #8188
Conversation
adamkotwasinski
force-pushed
the
kafka
branch
from
ce10704
to
8796af1
Compare
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
adamkotwasinski
force-pushed
the
kafka
branch
from
8796af1
to
8056d1e
Compare
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
adamkotwasinski
force-pushed
the
kafka
branch
from
af862dd
to
83d695d
Compare
…eption handling in broker-level filter Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
… tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
…sts and use these utils instead Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
|
@mattklein123 I have changed the test to be Given it's an integration test and it might run differently depending on machine speed, virtualization, other factors I can't predict today => WDYT about making it manual for now, and me working on making it become part of normal (automated) build in future. Now waiting for the build to go green. /wait |
Sure sounds good. Very excited for this to land and to figure out what we can build on top. |
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
|
/wait (will re-request review when I finally get it to green) |
…me during int tests Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
|
/wait-any |
|
Ready for re-review. |
| sha256 = "ae7a1696c0a0302b43c5b21e515c37e6ecd365941f68a510a7e442eebddf39a1", # 2.2.0-rc2 | ||
| strip_prefix = "kafka-2.2.0-rc2/clients/src/main/resources/common/message", | ||
| urls = ["https://github.com/apache/kafka/archive/2.2.0-rc2.zip"], |
There was a problem hiding this comment.
Congrats on shipping this @adamkotwasinski. I suspect this version lines up with when you started the PR. Kafka is now at 2.4.0 - https://github.com/apache/kafka/releases/tag/2.4.0. Are there any blockers on moving from the old RC release to the latest stable release?
There was a problem hiding this comment.
@moderation there should be none - I'll take a look as soon as I can
There was a problem hiding this comment.
@moderation work will be in #9582 ; I will need some more time to get the 2.4 running (the descriptor files used to generate the C++ code changed a little)
| strip_prefix = "kafka_2.12-2.2.0", | ||
| urls = ["http://us.mirrors.quenda.co/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz"], |
There was a problem hiding this comment.
Same for the server binary - http://us.mirrors.quenda.co/apache/kafka/2.4.0/
Description: Simple Kafka filter for broker. Grabs and decodes messages and updates metrics.
If message could not be recognised (the protocol used by client<->broker comms is higher than what's present in Envoy), we just pass through the payloads (IMHO it would be bad to have Envoy impact communication in this particular filter - this is not going to be possible in "mesh filter" as we we'd need).
In detail the change includes:
messaging_utilitiestest code library that's capable of making example payloads of various kinds (is going to be used for further tests).Relates to #2852
Risk Level: Low (Kafka code is unused right now)
Testing: automated tests, manual testing with real Kafka broker and client
Docs Changes: Kafka broker filter added
Release Notes: n/a