feat: Add kafka-go based kafka scaler #4801

sansmoraxz · 2023-07-15T18:15:54Z

Add experimental kafka scaler based on kafka-go apache-kafka.
Changes to e2e test cases for concurrent testing of multiple kafka scalers.
Upgrade kafka version used for e2e testing.

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
Commits are signed with Developer Certificate of Origin (DCO - learn more)
A PR is opened to update the documentation on (repo) (if applicable)

Fixes #4692 #3431

Relates to kedacore/keda-docs#1187

- kafka tests should run in their own namespace - each kafka test case can have their own kafka cluster Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

github-actions · 2023-07-15T18:16:07Z

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

🏖️ Over the summer, the response time will be longer than usual due to maintainers taking time off so please bear with us.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

Our contribution guide

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

pkg/scaling/scalers_builder.go

sansmoraxz · 2023-07-17T11:49:14Z

I was waiting for test results on how it goes performance wise in a production grade use case. Seems pretty chill so far.

Putting it up for review.

JorTurFer

Thanks a lot for this awesome contribution!!!
I have done a first review over majority of the code

CHANGELOG.md

pkg/scalers/kafka_x_scaler.go

pkg/scalers/kafka_x_scaler_test.go

pkg/scaling/scalers_builder.go

pkg/util/helpers.go

Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Souyama <souyama.debnath@atos.net>

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

- rename kafka-x to apache-kafka Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz · 2023-07-30T12:41:48Z

As @JorTurFer suggested, I made the required changes.

JorTurFer

LGTM!
Could you take a look @zroubalik ?

JorTurFer · 2023-08-15T11:10:59Z

/run-e2e kafka
Update: You can check the progress here

amcgreevy-r7 · 2023-08-22T11:36:17Z

Is there an update on when we think this change will be merged? Eagerly waiting on the ability to use IAM for auth.

JorTurFer · 2023-08-24T16:19:38Z

We are waiting @zroubalik 's review as our expert in kafka. We are in summer break but we will restore the usual speed in September, so I guess that this will be merged soon

zroubalik · 2023-09-11T17:01:55Z

/run-e2e kafka
Update: You can check the progress here

pkg/scalers/apache_kafka_scaler.go

pkg/scaling/scalers_builder.go

pkg/scalers/apache_kafka_scaler.go

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

- With summary of diference between kafka scaler implementation Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

zroubalik · 2023-09-14T16:18:44Z

/run-e2e kafka
Update: You can check the progress here

sansmoraxz · 2023-09-14T16:22:23Z

Just a heads up, although there were minimal changes needed to make this compatible with v2, haven't tested it with a cluster yet.

Should not break anything, but will get around to testing it by mid of next week.

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

zroubalik · 2023-09-15T11:31:05Z

/run-e2e kafka
Update: You can check the progress here

zroubalik · 2023-09-15T11:31:28Z

Just a heads up, although there were minimal changes needed to make this compatible with v2, haven't tested it with a cluster yet.

Should not break anything, but will get around to testing it by mid of next week.

Awesome, thanks for the great work! Keep us posted.

sansmoraxz · 2023-09-23T18:05:08Z

Everything's working as expected.

zroubalik

LGTM, great work! Thanks

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net> Signed-off-by: Souyama <souyama.debnath@atos.net> Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: anton.lysina <alysina@gmail.com>

sansmoraxz added 5 commits July 15, 2023 19:20

Move strimzi install to e2e test setup

71c9153

- kafka tests should run in their own namespace - each kafka test case can have their own kafka cluster Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add dependencies for kafka-go

19efb5b

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add scaler implementation for kafka-x

52187ef

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add vendor for dependencies

e37f295

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add kafka-go impl to changelog

ed6b0ae

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz force-pushed the kafka-go branch from 3662dfe to bd19584 Compare July 15, 2023 20:03

fix: static check issues

6b0c624

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz force-pushed the kafka-go branch from bd19584 to 6b0c624 Compare July 15, 2023 21:54

fix: test cases

8efb8b0

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz mentioned this pull request Jul 16, 2023

docs: new experimental Kafka scaler kedacore/keda-docs#1187

Merged

1 task

tomkerkhove reviewed Jul 17, 2023

View reviewed changes

pkg/scaling/scalers_builder.go Outdated Show resolved Hide resolved

sansmoraxz marked this pull request as ready for review July 17, 2023 11:49

sansmoraxz requested a review from a team as a code owner July 17, 2023 11:49

JorTurFer reviewed Jul 26, 2023

View reviewed changes

sansmoraxz and others added 4 commits July 27, 2023 12:02

Update CHANGELOG.md

18dab2c

Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Souyama <souyama.debnath@atos.net>

Rename kafka-x to apache-kafka

f341c05

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add unit tests for generic Contains function

ad21352

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Update CHANGELOG.md

d8a6d10

- rename kafka-x to apache-kafka Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz force-pushed the kafka-go branch from 7e28a36 to d8a6d10 Compare July 27, 2023 07:48

JorTurFer mentioned this pull request Jul 31, 2023

GSSAPI sasl mechanism for Kafka scaler #4836

Closed

JorTurFer reviewed Aug 15, 2023

View reviewed changes

JorTurFer assigned zroubalik Aug 15, 2023

jeevanragula reviewed Sep 12, 2023

View reviewed changes

pkg/scalers/apache_kafka_scaler.go Outdated Show resolved Hide resolved

jeevanragula reviewed Sep 12, 2023

View reviewed changes

pkg/scaling/scalers_builder.go Outdated Show resolved Hide resolved

zroubalik reviewed Sep 13, 2023

View reviewed changes

pkg/scalers/apache_kafka_scaler.go Show resolved Hide resolved

sansmoraxz added 4 commits September 14, 2023 20:54

Merge branch 'main' into kafka-go

f71c00c

Use aws sdk v2 for AWS MSK login

cc82a66

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

go mod vendor

562b2c3

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

Add header block for apache-kafka scaler

defd1ac

- With summary of diference between kafka scaler implementation Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz force-pushed the kafka-go branch 2 times, most recently from 5648ca4 to 4f87694 Compare September 14, 2023 20:46

Sort Changelog

fcf7276

Signed-off-by: Souyama Debnath <souyama.debnath@atos.net>

sansmoraxz force-pushed the kafka-go branch from bc010ba to fcf7276 Compare September 15, 2023 08:48

Merge branch 'main' into kafka-go

378c242

zroubalik approved these changes Sep 25, 2023

View reviewed changes

zroubalik merged commit b9f8d15 into kedacore:main Sep 25, 2023
18 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add kafka-go based kafka scaler #4801

feat: Add kafka-go based kafka scaler #4801

sansmoraxz commented Jul 15, 2023 •

edited

github-actions bot commented Jul 15, 2023

sansmoraxz commented Jul 17, 2023

JorTurFer left a comment

sansmoraxz commented Jul 30, 2023

JorTurFer left a comment

JorTurFer commented Aug 15, 2023 •

edited by github-actions bot

amcgreevy-r7 commented Aug 22, 2023

JorTurFer commented Aug 24, 2023

zroubalik commented Sep 11, 2023 •

edited by github-actions bot

zroubalik commented Sep 14, 2023 •

edited by github-actions bot

sansmoraxz commented Sep 14, 2023

zroubalik commented Sep 15, 2023 •

edited by github-actions bot

zroubalik commented Sep 15, 2023

sansmoraxz commented Sep 23, 2023

zroubalik left a comment

feat: Add kafka-go based kafka scaler #4801

feat: Add kafka-go based kafka scaler #4801

Conversation

sansmoraxz commented Jul 15, 2023 • edited

Checklist

github-actions bot commented Jul 15, 2023

sansmoraxz commented Jul 17, 2023

JorTurFer left a comment

Choose a reason for hiding this comment

sansmoraxz commented Jul 30, 2023

JorTurFer left a comment

Choose a reason for hiding this comment

JorTurFer commented Aug 15, 2023 • edited by github-actions bot

amcgreevy-r7 commented Aug 22, 2023

JorTurFer commented Aug 24, 2023

zroubalik commented Sep 11, 2023 • edited by github-actions bot

zroubalik commented Sep 14, 2023 • edited by github-actions bot

sansmoraxz commented Sep 14, 2023

zroubalik commented Sep 15, 2023 • edited by github-actions bot

zroubalik commented Sep 15, 2023

sansmoraxz commented Sep 23, 2023

zroubalik left a comment

Choose a reason for hiding this comment

sansmoraxz commented Jul 15, 2023 •

edited

JorTurFer commented Aug 15, 2023 •

edited by github-actions bot

zroubalik commented Sep 11, 2023 •

edited by github-actions bot

zroubalik commented Sep 14, 2023 •

edited by github-actions bot

zroubalik commented Sep 15, 2023 •

edited by github-actions bot