-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change kafka library to kafka-go #4692
Comments
LGTM! Will do this. /assign |
I was working on this. Perhaps we can collaborate. |
Just to put it out there, kafka-go does provide an interface for various sasl authentication mechanisms, you can check out here: https://github.com/segmentio/kafka-go/tree/main/sasl Implementing any new sasl based authentication mechanisms will be easier as we don't need to wait for the library vendor (segment) to add support for them. |
Cool, @sansmoraxz are you able to tell me if we can use the same configuration & settings with this library?
|
I am not sure if the |
Extremely sorry, I had no idea you were working on it.
Sure that's a great advantage!
Sure thing! If you are currently working on the implementation part or migrating current implementation from Sarama to Kafka-go. Let me know once you are done, I will add auth support for other SASL mechanism as well. Further I will take a look at Kafka-go, they have many auth providers missing. So maybe it'd be a good idea to add some in them too. |
My bad. Didn't mention it here before.
Sure, perhaps you can take a dig at Oauth. |
Quick update: I have implemented the core code changes. I have a question regarding the unit tests, viz for checking lag? How are we going to approach it? Can we use embeded docker container for this or is it gonna break something? For example: https://github.com/orlangure/gnomock |
@sansmoraxz I think we can give it a try, as discussed on Slack. Btw have you changed the scaler completely? I think that the safest option for us is to provide an alternative implementation and don't drop Sarama implementation (at least initially). |
Oops that I did. Need to rollback those. Yes it makes sense, to keep them seperate. |
Should we create a separate scaler i.e |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
/unstale |
Proposal
Switch the kafka client library to use Kafka-go in place of sarama
Use-Case
Kafka-go has support for MSK IAM which can help address #3431
In addition to this it has much cleaner api, better documented and much better resource utilization.
Is this a feature you are interested in implementing yourself?
Maybe
Anything else?
kafka-go doesn't provide mock objects so any tests that need to be done has to be made through a kafka server (perhaps a seperate container can be provisioned for this).
In addition to this the supported auth types include:
I guess Oauthbearer and GSSAPI is as of yet not supported in kafka-go.
The text was updated successfully, but these errors were encountered: