brod_group_coordinator gets blocked by assignments_revoked and not able to accept commit_offsets

[slashmili]

This issue was initially discussed in dashbitco/broadway_kafka#118

Hello,

We are using brod in broadway_kafka and we are facing a race condition situation.

brod_group_coordinator is asking to revoke the assignments, BroadwayKafka before returning to assignments_revoked waits until the on-going jobs are finished and tries to execute commit_offsets. However brod_group_coordinator gen_server is busy and is not able to accept any new message. Luckily commit_offsets has a timeout to avoid deadlock

sequenceDiagram
    brod_group_coordinator->>+BroadwayKafka: assignments_revoked
    BroadwayKafka->>brod_group_coordinator: commit_offsets
    brod_group_coordinator->>BroadwayKafka: {error, timetout}
    BroadwayKafka-->>-brod_group_coordinator: Done

Loading

But still all the commit_offsets are missed, which means they will be re consuming again. The timeout set on commit_offsets is 5 seconds, it means that BroadwayKafka might reply on assignments_revoked up to 20 seconds which in return triggers timeout on joining to a new generation and causes endless rebalancing.

My Questions:

1. Is there a way to commit the offset independent of brod_group_coordinator?
2. Is the approach we take in broadway_kafka correct? How other users of this library behave on assignments_revoked ?

[zmstone]

calling commit_offsets from subscriber has a known race condition.
Maybe call brod_group_coordinator:ack/5 instead?
the coordinator process will try its best to collect and commit the acked offsets.

[slashmili]

Thanks @zmstone.

As it's a known issue, we can find a workaround for time being.

But is it something that brod wants to improve? What if the commit offest was not tight to brod_group_coordinator ?

[slashmili]

I close this as it's a an expected behaviour