auto_commit not working in KafkaEx.stream/2

[gjaldon]

I'm using Kakfa 0.10.0 and have the following code:

for message <- KafkaEx.stream(@topic, partition) do
  process_message(pool, message)
end

The issue is that offsets are not being committed. I tried disabling the auto_commit option and then manually creating a worker and then manually committing the offset after the task but it still doesn't work. I based the offset_commit code on your test for the offset_commit function.

Also, process_message/2 above takes about half a minute to finish. In case, that matters.

What could be causing this problem?

[bjhaid]

@gjaldon sorry for the late response, have you tried KafkaEx.fetch do you have similar problems, I will attempt to reproduce sometimes today

[gjaldon]

@bjhaid no problem at all. I ended up getting the last offset from KafkaEx.fetch and passed the offset there to KafkaEx.stream as option so that it would start from that offset as a workaround.

[spscream]

try to use a named consumer group as a workaround, worked for us

[dantswain]

@gjaldon Can you provide a more complete reproduction case? I'd like to figure out what's going on here. Thanks!

[dantswain]

@bjhaid were you ever able to reproduce this?

[bjhaid]

I didn't try to 🤦‍♂️