#239 integrate sarama consumer group #260

frairon · 2020-07-08T11:39:22Z

integrate sarama consumer group
remove sarama-cluster dependency

Co-authored-by: R053NR07 <jan.bickel@lovoo.com>

updated readme for configuration, added changelog

Open Storage in PartitionTable when performing Setup

return trackOutput if stats are nil

added lots of godoc fixed many linter errors added Open call when creating storage

added lots of godoc, fixed many linter errors, added Open call when c…

stats-tracking improved

added strings to streams helper

Add Backoff when restartable CatchupForever is executed

Panic documentation & Migration guide

…wn the signal/observers, created example

248 connection state

readme example

fkarakas · 2020-11-19T16:00:45Z

Hello @frairon,
I have a question : Why did u remove confluent lib for sarama ?
Because the official library is maintained and updated by confluent so more "safe" regarding its maintenance in the future ...
thanks

frairon · 2020-11-19T16:06:43Z

Hi @fkarakas, you mean the support for the C implementation of kafka (I think it was based on librdkafka?), right?

The thing was, that this implementation follows a completely different logic to rebalance and receive messages. In the "old" goka version, we had to do quite some magic to support both sarama and librdkafka, so it was very hard to understand, maintain, and it behaved flaky in an unstable kafka cluster. So we decided to simplify the logic and get rid of it for now.
If this is something that people still need, I think we can somehow get it working again by trying to bridge its behavior to the new sarama logic of the consumer group. But haven't looked into it at all so far.
If you are willing to work on that, PRs are always welcome :D

fkarakas · 2020-11-19T16:12:56Z

Thanks, if i understand correctly it was difficult to make an abstraction in the goka API which can both work with confluent and sarama because internally they work differently

frairon · 2020-11-19T16:16:15Z

Yep exactly, and the point of the refactoring was to simplify things, get rid of sarama-cluster. Are you using the confluent api?

fkarakas · 2020-11-19T16:28:02Z

Yes we are using the confluent lib, as you mentioned it is based on the C librdkafka so it is not a full GO lib which can be a stopper for go purist, but we decided to use it because it is confluent official lib for GO so we think it is "safer" regarding its maintenance/upgrade in the future ...
And in kafka, a big part of the "logic" is implemented in the client side instead of the server side, so it was another +1 in the choice of confluent-kafka-go. This was our conclusion.

frairon · 2020-11-19T16:41:03Z

That sounds like a reasonable choice. So is this a blocker for you to upgrade to a new goka version? And would you have the resources to look into it?

R053NR07 · 2020-12-04T10:35:16Z

Hey @fkarakas,
as you said the "pure GO" implementation was the main driver for the decision. CGO always was and is a problem for us, we had to take extra effort to use goka since this was a requirement. Also goka internally we focused mainly on sarama and supporting confluentlib would have been extra effort. We decided to focus on sarama since it is widely used in the go world for working with kafka and we did not see any major limitations or drawbacks when using it.

Regarding the support. Sure librdkafka may get longer support but even if sarama would be deprecated today we would be able to integrate and switch to librdkafka; for us the effort would be almost the same then designing goka with interchangeable kafka libs.

But still. If that's a blocker for you and others we can talk about changing this in future releases.

frairon and others added 28 commits March 19, 2020 13:52

Co-authored-by: frairon <franz.eichhorn@lovoo.com>

aa4b10f

Co-authored-by: R053NR07 <jan.bickel@lovoo.com>

bugfix in shutdown/rebalance: correctly closing joins

b997692

run update/request/response stats in own goroutine

1e92aeb

fix rebalancing by adding a copartitioning rebalance strategy

35fcae1

updated readme for configuration, added changelog

3072453

Merge pull request #240 from lovoo/readme-changelog

bf2c5b4

updated readme for configuration, added changelog

Open Storage in PartitionTable when performing Setup

95ad600

Merge pull request #242 from lovoo/open_storage_in_setup

bf1c513

Open Storage in PartitionTable when performing Setup

return trackOutput if stats are nil

e1f8fd9

Merge pull request #244 from lovoo/fix-trackOutput

859992c

return trackOutput if stats are nil

view.get fixed for uninitialized view

38b6856

added lots of godoc fixed many linter errors added Open call when creating storage

Merge pull request #243 from lovoo/open_storage_in_setup

ed7dcbc

added lots of godoc, fixed many linter errors, added Open call when c…

context stats tracking: use queueing mechanism to avoid race conditions

3ab367a

Merge pull request #245 from lovoo/bugfix-track-stats

1eaa22f

stats-tracking improved

Add simpleBackoff and add backoff options for processor and view

bc71a7b

added strings to streams helper

b281011

Merge pull request #250 from lovoo/streams-helper

21558b3

added strings to streams helper

Merge pull request #246 from lovoo/add-retry-backoff

40abaea

Add Backoff when restartable CatchupForever is executed

#249 view example

f818f57

issue #249: migration guide, #241 panic documentation of context

1b7c0e2

Merge pull request #252 from lovoo/249-migration-guide

f7ae456

Panic documentation & Migration guide

#248 exposing partition table's connection state to view

404bc24

Migration: describe offsetbug

3a1e326

partition_table: implement autoreconnect in recovery mode

40b650b

bugfix goroutine-leak in statsloop in partition table

bd5bf97

#248: refactored state merger, bugfix race condition when shutting do…

f47ae9c

…wn the signal/observers, created example

bugfix partition state notification

dc77c63

Merge pull request #256 from lovoo/248-connection-state

f79a6bf

248 connection state

R053NR07 previously approved these changes Jul 9, 2020

View reviewed changes

remove change in example, updated changelog

8075d3c

frairon dismissed R053NR07’s stale review via 8075d3c July 9, 2020 11:40

frairon and others added 3 commits July 9, 2020 13:53

fix readme example and add readme as example

e0b0b7a

restore 1-simplest fix, remove readme-example

812812d

Merge pull request #261 from lovoo/readme-example-fix

e6bac0e

readme example

R053NR07 approved these changes Jul 9, 2020

View reviewed changes

frairon merged commit 075bc83 into master Jul 9, 2020

frairon deleted the consumer-group branch July 9, 2020 12:05

chin8628 mentioned this pull request Sep 22, 2021

Is Goka still support Confluent-Kafka client? #349

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#239 integrate sarama consumer group #260

#239 integrate sarama consumer group #260

frairon commented Jul 8, 2020

fkarakas commented Nov 19, 2020 •

edited

Loading

frairon commented Nov 19, 2020

fkarakas commented Nov 19, 2020 •

edited

Loading

frairon commented Nov 19, 2020

fkarakas commented Nov 19, 2020

frairon commented Nov 19, 2020

R053NR07 commented Dec 4, 2020 •

edited by frairon

Loading

#239 integrate sarama consumer group #260

#239 integrate sarama consumer group #260

Conversation

frairon commented Jul 8, 2020

fkarakas commented Nov 19, 2020 • edited Loading

frairon commented Nov 19, 2020

fkarakas commented Nov 19, 2020 • edited Loading

frairon commented Nov 19, 2020

fkarakas commented Nov 19, 2020

frairon commented Nov 19, 2020

R053NR07 commented Dec 4, 2020 • edited by frairon Loading

fkarakas commented Nov 19, 2020 •

edited

Loading

fkarakas commented Nov 19, 2020 •

edited

Loading

R053NR07 commented Dec 4, 2020 •

edited by frairon

Loading