cannot run multiple consumers in the same group

[joy4eg]

I have kafka 2.3.0, and want to scale my application horizontally.
For example:

	g := goka.DefineGroup("my-group",
		goka.Input("input", new(messageCodec), handler),
		goka.Output("output", new(messageCodec)),
	)

	p, err := goka.NewProcessor(addr, g)

But when I tried to run the second instance I'm getting the following errors (like try to run examples/1-simplest twice):

2019/10/18 10:24:17 error running processor: Errors:
        * kafka error: kafka server: The provided member is not known in the current generation.
        * error closing consumer: Failed to close consumer: kafka server: The provided member is not known in the current generation.

Have I missed something ?
Thanks!

[frairon]

Hi Yaroslav,
indeed we encountered this error a couple of times, but this was mostly caused by issues with the cluster itself.
Assuming your cluster is stable during the error I suspect there could be two other causes:
(1) goka/sarama doesn't go well with kafka 2.3.0. It's not really tested yet since we're still on an older version and it still relies on sarama-cluster, which is outdated by Sarama's consumer group implementation.
(2) there's a problem within the code in the handler callback. Maybe have a look at this issue IBM/sarama#1310 to check if that applies to your case and could be fixed by adjusting the timeouts.

I'm afraid reason (1) is more likely and harder to fix. To make sure that's not the issue, maybe try running your code against an older version of Kafka?

Hope that helps.
Let me know what the progress is or if/how you fixed the issue.
Cheers!

[joy4eg]

Hey,

Thanks for the quick reply!

I repeated my test (with examples/1-simplest) using kafka 2.1.1 and it works well.
But I wonder what could went wrong with a few minor updates ...

Anyway, thank you.

[frairon]

@joy4eg, as the issue seems to be fixed for you I'd like to close the issue. Just wanted to let you know that we're working on an improved version of goka to support newer kafka versions. Feel free to try branch consumer-group or tag v0.9.0-beta2 and let me know if you're having issues.
Thanks!