Python 3.12.0 compatibility issue

[pawel-swiecki-saucelabs]

Describe the bug
There is a Python 3.12.0 compatibility issue of aiokafka's dependency, namely kafka-python. The issue has been reported here and there already is a fix waiting to be reviewed here. (By the looks of open issues and open PRs it seems the library is not actively maintained.)

Expected behaviour
No ModuleNotFoundError: No module named 'kafka.vendor.six.moves' when importing aiokafka on Python 3.12.0.

Environment

• Python version: 3.12.0
• aiokafka version: 0.8.1
• kafka-python version: 2.0.2
• Kafka Broker version: not relevant

Reproducible example

$ cat Dockerfile
FROM python:3.12-bullseye
RUN python -m venv venv
RUN ./venv/bin/pip install aiokafka
ENTRYPOINT ["/venv/bin/python", "-c", "import aiokafka"]

$ docker run -it $(docker build -q .)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.12/site-packages/aiokafka/__init__.py", line 3, in <module>
    from .abc import ConsumerRebalanceListener
  File "/venv/lib/python3.12/site-packages/aiokafka/abc.py", line 2, in <module>
    from kafka import ConsumerRebalanceListener as BaseConsumerRebalanceListener
  File "/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[Arthur264]

+1

[SpudInNZ]

Also waiting on this, is there any ETA? Thanks.

Edit: just realised the issue is with underlying Kafka package: dpkp/kafka-python#2401

[kPsarakis]

kafka-python is not actively maintained or getting any of the new features (deprecated) dpkp/kafka-python#2290. We should help the developers of aiokafka towards the standalone package goal as discussed here #915

[ods]

@kPsarakis I'm at finish line of merging kafka-python's code into aiokafka. Are you willing to review test it?

[kPsarakis]

@ods Yes, let me know how I can help.

[ods]

@kPsarakis #932

[jainal09]

@ods I saw that your PR #932 was merged. But, I see that I am still getting this error with the latest merge code which is not dependent on the kafka-python project

[ods]

But, I see that I am still getting this error with the latest merge code which is not dependent on the kafka-python project

Do you have kafka-python installed and import it from your code directly? Please show traceback with the latest code from master.

[AntonOfTheWoods]

Looks like this is good to go - any chance of a release?

[ods]

Looks like this is good to go - any chance of a release?

There are more issues to fix before release. One is the problem with python-snappy (replacing it with cramjam is in process), and another is moving away from setup.py (not so simple due to Cython).

[ods]

One more step towards new release with Python 3.12 support: #940. It would be very helpful if those using aiokafka with Snappy and/or ZStandard compression check it with their applications.

[SpudInNZ]

Speaking for us @ods it's much easier for us to test a new version if you release to pypi as a beta.

[AntonOfTheWoods]

Yep, and the quicker this gets out, the higher the likelihood kafka-python won't get a 3.12 release, and we can let it die the death it deserves!

[ods]

Ok, here is pre-release

[ods]

And one more, 0.9.0rc1.

[ods]

Let's move this discussion to the topic for the prerelease: #948