Updating from kafka-1.3.5 to kafka-python-1.4.6 leads to "TypeError: 'NoneType' object is not iterable"

[eeepmb]

Hi guys,
I ran into the above mentioned TypeError after upgrading the Kafka Library.

Below you can find the condensed script and a extract of my cli. I tried to go back and forth with both Kafka Lib (1.3.5 / 1.4.6) and Python (3.6.5 / 3.7.3) versions which I tried to display with the cli extract. (Side note: When I use Python 3.7 with Kafka 1.3.5 I run into issue #1566 because async is a reserved keyword since this version)
I work with macOS Mojave 10.14.5.

I'm not sure if this is really a bug but since my script worked before I think it is at least worth mentioning.

The script:

from kafka import KafkaProducer, KafkaConsumer, TopicPartition
from kafka.errors import KafkaError

bsServers=['localhost:9092']

outputTopic='test'
outputGroupID='testGroup'

outputConsumer = KafkaConsumer(
                         group_id=outputGroupID,
                         bootstrap_servers=bsServers,
                         auto_offset_reset='latest',
                         enable_auto_commit=True)

PARTITIONS = []
for partition in outputConsumer.partitions_for_topic(outputTopic):
    PARTITIONS.append(TopicPartition(outputTopic, partition))

partitions = outputConsumer.end_offsets(PARTITIONS)

print("Partitions and offsets:", partitions)

outputConsumer.assign(partitions)

print("Done.")

CLI extract:

$ pip3 list
Package           Version 
----------------- --------
kafka-python      1.4.6   

$ python3 --version
Python 3.7.3

$ python3 script.py 
Traceback (most recent call last):
  File "script.py", line 28, in <module>
    for partition in outputConsumer.partitions_for_topic(outputTopic):
TypeError: 'NoneType' object is not iterable

$ pip3 uninstall kafka-python
Uninstalling kafka-python-1.4.6:
  Would remove:
    /usr/local/lib/python3.7/site-packages/kafka/*
    /usr/local/lib/python3.7/site-packages/kafka_python-1.4.6.dist-info/*
Proceed (y/n)? y
  Successfully uninstalled kafka-python-1.4.6

$ pip3 install kafka
Collecting kafka
  Using cached https://files.pythonhosted.org/packages/21/71/73286e748ac5045b6a669c2fe44b03ac4c5d3d2af9291c4c6fc76438a9a9/kafka-1.3.5-py2.py3-none-any.whl
Installing collected packages: kafka
Successfully installed kafka-1.3.5

$ python3 script.py 
Traceback (most recent call last):
  File "script.py", line 1, in <module>
    from kafka import KafkaProducer, KafkaConsumer, TopicPartition
  File "/usr/local/lib/python3.7/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.producer import KafkaProducer
  File "/usr/local/lib/python3.7/site-packages/kafka/producer/__init__.py", line 4, in <module>
    from .simple import SimpleProducer
  File "/usr/local/lib/python3.7/site-packages/kafka/producer/simple.py", line 54
    return '<SimpleProducer batch=%s>' % self.async
                                                  ^
SyntaxError: invalid syntax

$ pip3 uninstall kafka
Uninstalling kafka-1.3.5:
  Would remove:
    /usr/local/lib/python3.7/site-packages/kafka-1.3.5.dist-info/*
    /usr/local/lib/python3.7/site-packages/kafka/*
Proceed (y/n)? y
  Successfully uninstalled kafka-1.3.5

$ brew switch python3 3.6.5_1
Cleaning /usr/local/Cellar/python/3.6.5_1
Cleaning /usr/local/Cellar/python/3.7.3
25 links created for /usr/local/Cellar/python/3.6.5_1

$ pip3 install kafka-python
Collecting kafka-python
  Using cached https://files.pythonhosted.org/packages/82/39/aebe3ad518513bbb2260dd84ac21e5c30af860cc4c95b32acbd64b9d9d0d/kafka_python-1.4.6-py2.py3-none-any.whl
Installing collected packages: kafka-python
Successfully installed kafka-python-1.4.6

$ python3 script.py 
Traceback (most recent call last):
  File "script.py", line 28, in <module>
    for partition in outputConsumer.partitions_for_topic(outputTopic):
TypeError: 'NoneType' object is not iterable

$ pip3 uninstall kafka-python
Uninstalling kafka-python-1.4.6:
  Would remove:
    /usr/local/lib/python3.6/site-packages/kafka/*
    /usr/local/lib/python3.6/site-packages/kafka_python-1.4.6.dist-info/*
Proceed (y/n)? y
  Successfully uninstalled kafka-python-1.4.6

$ pip3 install kafka
Collecting kafka
  Using cached https://files.pythonhosted.org/packages/21/71/73286e748ac5045b6a669c2fe44b03ac4c5d3d2af9291c4c6fc76438a9a9/kafka-1.3.5-py2.py3-none-any.whl
Installing collected packages: kafka
Successfully installed kafka-1.3.5

$ python3 script.py 
Partitions and offsets: {TopicPartition(topic='test', partition=0): 0}
Done.

Thanks in advance

[dpkp]

partitions_for_topic is currently non-blocking and will return None if the consumer has not received topic/partition metadata yet. A workaround is to call consumer.topics() first -- this is blocking and will request the full topic/partition list from the cluster.

[esparzak]

@dpkp Is there a similar workaround for the producer? (As it does not have the topics() method) We are seeing the same issue when sending a message to a topic with the producer