Sarama ConsumerGroup Update

[frairon]

In this issue we want to announce the upcoming release of a new and majorly refactored version of Goka.
Spoiler: there will be breaking changes, but nothing that can't be fixed with simple text-replace in your projects. So no worries. We'll provide a small migration guide, but most of the things will be obvious anyway.

So what are the major changes:

• Removed sarama-cluster, replaced with sarama's ConsumerGroup
• remove package kafka, move remaining files to root
• introduce go mod
• Sorry, removed support for librdkafka

We'll tag the version shortly so it can be tested. Let us know if there are any issues so we can quickly react.

[mattburman]

Hi @frairon, I've been evaluating the latest beta changes and writing a wrapper around it for our own usage. Previously we had some logic around view.Run which implemented retry behaviour with an exponential backoff.

Re-testing this logic with the new versions shows that view.Run no longer returns like before and it's reconnecting with its own logic. That's great that the retries are within Goka now and we can remove some of our own logic.

I think I've narrowed it down to view.Run now looping over each partition and calling CatchupForever: v0.1.4...v0.9.0-beta1#diff-6202d9bde3a4cbd1c7a28821646249d4
(GitHub seems to have a bug with links with hashes so try v0.1.4...v0.9.0-beta1 with #diff-6202d9bde3a4cbd1c7a28821646249d4 at the end or just go to view.go)

This is great, but it would be good to have some kind of control over the retry behaviour. We previously has view.Run wrapped with a configurable backoff using https://github.com/jpillora/backoff. What are your thoughts on this? For now I am OK with removing our custom logic and keeping the default retry behaviour implemented in Goka. I would still welcome any thoughts on making the behaviour configurable, however.

Additionally, it would be nice to expose the view state: possibly ViewStateIdle, ViewStateCatchUp, or ViewStateRunning? Ideally I want to be able to differentiate between being behind and a connection cant currently be established.

I would suggest any migration guide includes information about this change to view.Run because it took me a while to realise our previous backoff behaviour was no longer running.

[mattburman]

I have also found the view.Recovered method returns true whilst it is struggling to connect to kafka. Is this intended? I need some way of determining whether the app is connected to kafka for metrics. It looks like partitionStats.Status also returns 4 (PartitionRunning) when no connection to kafka can be established. Stalled returns false too

[mattburman]

I have also found the view.Recovered method returns true whilst it is struggling to connect to kafka. Is this intended? I need some way of determining whether the app is connected to kafka for metrics. It looks like partitionStats.Status also returns 4 (PartitionRunning) when no connection to kafka can be established. Stalled returns false too

Looks like #246 might provide capability to track connection status outside of goka, but in a bit of a hacky way.

• store a Connected boolean on the struct implementing goka.Backoff
• append the struct to a list of all the backoffs in the implementation of BackoffBuilder
• set to false when Duration() is called
• set to true when Reset() is called

Then some status checker can possibly loop over all of these backoff structs and check their status independently.

[frairon]

@mattburman you're on fire here :D.
Let's sum up the issues (and actually create different issues for them).
Backoff-Behavior is discussed and handled in #247.
I also just created an issue for the view state/connection state #248 and one for the migration guide #249.

[frairon]

The PR is merged, the version is released (v1.0.0). I'll close the issue, feel free to open new ones for possible bugs/new issues