Must call Poseidon::ConsumerGroup#close when you're done with ConsumerGroup!

[pedros007]

Problem: When I instantiate two consumer groups without calling #close, the second ConsumerGroup does not ingest any messages. Here's an example:

require 'poseidon_cluster'

producer = Poseidon::Producer.new(["localhost:9092"], "worlds best producer")
m = Poseidon::MessageToSend.new("test", "the contents of my message")
producer.send_messages( [m] )
producer = nil


[1,2].each do |iteration|
  puts "Fetching... on iteration #{iteration}"
  consumer = Poseidon::ConsumerGroup.new("world greatest consumer 1", ["localhost:9092"], ["localhost:2181"], "test")
  consumer.fetch commit: false do |partition, messages|
    puts "Consumer #1 fetched #{messages.size} from #{partition}."
  end
  consumer=nil
end

Output is:

Fetching... on iteration 1
Consumer #1 fetched 1 from 0.
Fetching... on iteration 2

but I expected to see

Fetching... on iteration 1
Consumer #1 fetched 1 from 0.
Fetching... on iteration 2
Consumer #1 fetched 1 from 0.

The only way I can get the expected output is if I call consumer.close() in each iteration. This was really surprising and not documented in the README or examples.

Perhaps when consumer goes out of scope, you could leverage define_finalizer to call close to prevent this from happening?

[pedros007]

In the meantime, I will probably wrap my code as follows:

begin
  consumer = Poseidon::ConsumerGroup.new("world greatest consumer 1", ["localhost:9092"], ["localhost:2181"], "test")
  # do whatever I need to do with the consumer object.
ensure
  consumer.close
end

[dim]

Could you please submit a PR with a test?

[mensfeld]

+1 it seems I have the same issue

[daluu]

I decided to follow the recommendation here just in case. For some reason I get error trying to call consumer.close() in ensure block:

... in ensure in get_current_offset': undefined method 'close' for nil:NilClass (NoMethodError)

I'm using poseidon_cluster 0.3.2 with ruby 2.2.1

[mensfeld]

@daluu could you prepare a test case?

[daluu]

It's been a while and I stopped working with poseidon cluster. But on a quick check/test, it seems my issue does not occur now. Not sure what happened before.

[mensfeld]

@daluu I cannot reproduce as well. Could you please close the issue then?