Always request metadata when set empty topics list for KafkaClient

[laz2]

KafkaConsumer can not resubscribe to new pattern without this fix because setting empty topics list always does not request metadata.

Does KafkaClient.set_topics need additional kwarg for request metadata?

PS. KafkaConsumer.unsubcribe calls set_topics with empty list. Is this problem?

[dpkp]

Perhaps a simpler change would be to add a call to self._client.cluster.request_update() to KafkaConsumer.subscribe() whenever it is called with a pattern. This is basically how the java client handles this case.

[laz2]

So may be remove call request_update() and future return value from KafkaClient.set_topics?

[dpkp]

I dont think we need to change the set_topics code at all if the main issue is that pattern subscriptions are not requesting a metadata update on subscribe(). If that is the problem, why not request a metadata update directly from subscribe() ?

[laz2]

Is error Broken pipe problem?

[dpkp]

no, unrelated error during test cleanup / client shutdown

[laz2]

Any suggestions about this problem?

[jeffwidman]

What was the underlying problem here?

I'm a little unclear what this fixes and trying to track down a bug that might be related. Did this apply to a consumer that subscribes to a valid regex pattern and receives no topics in reply because none exist yet?

[dpkp]

jeff: the issue would apply to consumers that have already fetched metadata (perhaps by initially subscribing to a topic list). So:

consumer.subscribe(topics=['foo'])
consumer.poll()
consumer.subscribe(pattern='foo*')
consumer.poll()

Pattern subscriptions in general will only get "updates" every metadata_max_age_ms when the client refreshes metadata to get new topic information -- or if there is a broker restart etc that triggers an immediate refresh.