This repository has been archived by the owner on Apr 11, 2024. It is now read-only.
feat: add first poc for a Kafka proxy #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smoya
commented
Jun 23, 2021
| @@ -0,0 +1,47 @@ | |||
| linters: | |||
There was a problem hiding this comment.
smoya
commented
Jun 23, 2021
| @@ -0,0 +1,57 @@ | |||
| # Makefile based on https://gist.github.com/thomaspoignant/5b72d579bd5f311904d973652180c705 | |||
There was a problem hiding this comment.
smoya
commented
Jun 23, 2021
| gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect | ||
| ) | ||
|
|
||
| replace github.com/grepplabs/kafka-proxy => github.com/smoya/kafka-proxy v0.2.9-0.20210623125118-a94cf71a065c |
There was a problem hiding this comment.
Here is the replace directive in order to use my forked version of grepplabs/kafka-proxy
smoya
commented
Jun 23, 2021
| @@ -8,8 +8,11 @@ | |||
| # Test binary, built with `go test -c` | |||
There was a problem hiding this comment.
smoya
commented
Jun 23, 2021
| @@ -0,0 +1,118 @@ | |||
| package main | |||
There was a problem hiding this comment.
This file is actually the code of the app :)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds contains the first POC of a Kafka Proxy that validates messages. As a poc, there has no real validation logic behind but the functionality is ready to make it happen: Right now it only prints a log line whenever a message is literally the string
invalid message.The PR also includes a
docker-compose.ymland aDockerfileso we can run a one node Kafka cluster and do some testing. Readme update is pending.If you wanna play in the meantime, you can just run
docker compose upand produce/consume messages by:Consume:
docker compose exec broker /bin/kafka-console-consumer --topic demo --from-beginning --bootstrap-server proxy:28002Produce:
docker compose exec broker /bin/kafka-console-producer --topic demo --bootstrap-server proxy:28002and start writing messages (stdin).As notes for developers, I managed to completely turn off my old POC (yeah, never pushed but I can share it with you as I have it locally) where I had to manage almost all the protocol handling manually (yeah, dealing with bits and bytes) and which was only supporting a particular Kafka request protocol version.
Now I'm using https://github.com/grepplabs/kafka-proxy which it has almost everything we need. In fact I had to fork it and tune it up a bit (Adding support for request handlers per Key. See the fork here. I plan to create a PR to the main project once I have more time to improve those changes I did.
I'm also using parts of the code of https://github.com/Shopify/sarama project for decoding and validating the
ProduceRequest. I could not use the project directly because some of the interfaces I needed are private. I opened an issue few days ago regarding this: IBM/sarama#1967.One of the biggest achievements on using https://github.com/grepplabs/kafka-proxy is that it supports most of the AUTH mechanisms and also all the Kafka protocol request versions (YEEEEI!).
Next steps
How to review this PR
Even though this is a POC, most of this code could be used as the foundation of the first versions of the gateway.
That means you should review it as any other PR. Some things that were not included on this PR with the idea of doing it in another one are:
main.goSome things that were explicitly excluded from this PR were:
/kafka)Related issue(s)
asyncapi/shape-up-process#70