Clearly document that KafkaConsumer is not thread safe

[ecksun]

The question is basically in the title.

On the one hand, we had MultiprocessConsumer which I assume must be, but it has been deprecated in favor of KafkaConsumer, which leads me to think that KafkaConsumer also is thread safe.

On the other hand I just got this error using it:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "code/node_list/reaction.py", line 27, in run
  File "code/env/local/lib/python3.5/site-packages/event_consumers/event_consumer.py", line 28, in __iter__
    for message in self.consumer:
  File "code/env/local/lib/python3.5/site-packages/kafka/consumer/group.py", line 964, in __next__
    return next(self._iterator)
  File "code/env/local/lib/python3.5/site-packages/kafka/consumer/group.py", line 904, in _message_generator
    self._client.poll(timeout_ms=poll_ms, sleep=True)
  File "code/env/local/lib/python3.5/site-packages/kafka/client_async.py", line 525, in poll
    responses.extend(self._poll(timeout, sleep=sleep))
  File "code/env/local/lib/python3.5/site-packages/kafka/client_async.py", line 578, in _poll
    response = conn.recv()  # Note: conn.recv runs callbacks / errbacks
  File "code/env/local/lib/python3.5/site-packages/kafka/conn.py", line 607, in recv
    response = self._recv()
  File "code/env/local/lib/python3.5/site-packages/kafka/conn.py", line 654, in _recv
    raise Errors.KafkaError('this should not happen - are you threading?')
kafka.errors.KafkaError: KafkaError: this should not happen - are you threading?

Which suggest it is not.

Edit: this example also clearly indicates that using threading should be fine

Can I safely use KafkaConsumer in a threaded environment?

[dpkp]

No. KafkaConsumer is not suitable for sharing across threads.

…

On May 18, 2017 7:27 AM, "ecksun" ***@***.***> wrote: The question is basically in the title. On the one hand, we had MultiprocessConsumer which I assume must be, but it has been deprecated in favor of KafkaConsumer, which leads me to think that KafkaConsumer also is thread safe. On the other hand I just got this error using it: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "code/node_list/reaction.py", line 27, in run File "code/env/local/lib/python3.5/site-packages/event_consumers/event_consumer.py", line 28, in __iter__ for message in self.consumer: File "code/env/local/lib/python3.5/site-packages/kafka/consumer/group.py", line 964, in __next__ return next(self._iterator) File "code/env/local/lib/python3.5/site-packages/kafka/consumer/group.py", line 904, in _message_generator self._client.poll(timeout_ms=poll_ms, sleep=True) File "code/env/local/lib/python3.5/site-packages/kafka/client_async.py", line 525, in poll responses.extend(self._poll(timeout, sleep=sleep)) File "code/env/local/lib/python3.5/site-packages/kafka/client_async.py", line 578, in _poll response = conn.recv() # Note: conn.recv runs callbacks / errbacks File "code/env/local/lib/python3.5/site-packages/kafka/conn.py", line 607, in recv response = self._recv() File "code/env/local/lib/python3.5/site-packages/kafka/conn.py", line 654, in _recv raise Errors.KafkaError('this should not happen - are you threading?') kafka.errors.KafkaError: KafkaError: this should not happen - are you threading? Which suggest it is not. Can I safely use KafkaConsumer in a threaded environment? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1105>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetEZ_COui1-C0MqIFGeW2CwNaGBD-ks5r7FMdgaJpZM4NfTZc> .

[ecksun]

No. KafkaConsumer is not suitable for sharing across threads.

Alright, if I interpret that literally thats not a problem in my case (nor in the example). As both I and the example I linked have one consumer per thread. Is that also an unsuitable setup?

[dpkp]

Thread-local KafkaConsumer should work. I wouldn't recommend it for other reasons, but it should not crash.

…

On May 18, 2017 8:56 AM, "ecksun" ***@***.***> wrote: No. KafkaConsumer is not suitable for sharing across threads. Alright, if I interpret that literally thats not a problem in my case (nor in the example). As both I and the example I linked have one consumer per thread. Is that also an unsuitable setup? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetLbTDltrMv4jZSfGMcOiRbbibiMyks5r7GoggaJpZM4NfTZc> .

[ecksun]

Thread-local KafkaConsumer should work. I wouldn't recommend it for other
reasons, but it should not crash.

What reasons are those? What would you recommend?

[dpkp]

I'd recommend multiprocessing for running parallel consumers. Python threading is not useful given GIL.

…

On May 18, 2017 9:09 AM, "ecksun" ***@***.***> wrote: Thread-local KafkaConsumer should work. I wouldn't recommend it for other reasons, but it should not crash. What reasons are those? What would you recommend? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetJrZmWYhkBoKlsaFWcuZK7M0GzPIks5r7G1DgaJpZM4NfTZc> .

[jeffwidman]

Actually, re-opening as this is a consistent question on the issue tracker, implying we should more explicitly address this in the readme and/or docs.

[ecksun]

I'd recommend multiprocessing for running parallel consumers. Python
threading is not useful given GIL.

Yes, I actually rewrote all my stuff to use multiprocessing, turns out the instrumentation library I use (prometheus) doesn't support that :(

Actually, re-opening as this is a consistent question on the issue tracker, implying we should more explicitly address this in the readme and/or docs.

Yes, I was going to suggest that.

Both detailing in the docs that no, it is not thread safe (unlike the producer), as well as the recommendation for multiprocessing.

[ecksun]

I created a small PR for it, consider it a draft and feel free to do anything with it, I just thought I would provide something to start with.

[dpkp]

Thanks for the PR w/ documentation updates. Merged!