Handle EOF partition error properly in Consumer#each

[jurriaan]

When trying this library I got the following error:

/app/vendor/bundle/ruby/2.4.0/gems/rdkafka-0.3.3/lib/rdkafka/consumer.rb:186:in `poll': Broker: No more messages (partition_eof) (Rdkafka::RdkafkaError)
	from /app/vendor/bundle/ruby/2.4.0/gems/rdkafka-0.3.3/lib/rdkafka/consumer.rb:207:in `block in each'
	from /app/vendor/bundle/ruby/2.4.0/gems/rdkafka-0.3.3/lib/rdkafka/consumer.rb:206:in `loop'
	from /app/vendor/bundle/ruby/2.4.0/gems/rdkafka-0.3.3/lib/rdkafka/consumer.rb:206:in `each'

I think the #each method should handle these errors (ignore them, or pass these notifications to the block). Currently the each method itself raises an exception when you reach the end of the partition which is probably not what people expect from this library.

[thijsc]

You can configure this behaviour with enable.partition.eof. I agree that it's counterintuitive that this happens with each by default.

I see a few options:

• each could handle the exception and just keep going
• each could handle the exception and yield some kind of special eof message
• We could set enable.partition.eof to true by default

I think probably handling the exception makes the most sense. Not sure what is the nicest way to handle this, will think about it. What do you think?

[jurriaan]

I'm currently writing a Kafka Streams-like processor where I want to store the state in a compacted topic and read the state back into memory on start for which I need to know when I'm done. This could be accomplished by this special eof message. For most processors however this EOF messages is not needed so this should be configurable IMO.

[thijsc]

It is configurable already with enable.partition.eof, which is true by default in librdkafka.

[jurriaan]

@thijsc Ah yes, of course, there is no need for a configuration option in this library then, when the library receives an eof 'error' message it should yield some kind of special eof message. The enable.partition.eof flag can then be used to adjust this behaviour.

[thijsc]

Indeed. This issue is still open because I want to make the error handling nicer, but the configuration is good as is.

[thijsc]

By the way: If you want complete control over this you're probably better off calling poll in a loop.

[thijsc]

I'm still not sure how to make this better than it is, so decided to add a note in 45055f1 and close this. If anyone has good ideas about this one feel free to reopen.