[QUESTION] Future development and maintenance of aiokafka

[amotl]

Dear @asvetlov and @tvoinarovskyi,

first things first: Thanks a bunch for conceiving and maintaining aiokafka during the last years. You know who you are.

Now, I am humbly asking how you are looking at the future of this library. I see that #628 has been fixed by @iamsinghrajat the other day within #629 and #659 has been fixed by @dkilgore90 within #660 (thanks for these!). Also, @tvoinarovskyi already started working on Python 3.8 support on behalf of the fixpy38 branch with 946bbe9.

While #629 has been merged already and made it into a v0.6.1.dev0 pre-release, it did not make it into an official release available on PyPI. On the other hand, #660 is still open and the work on supporting Python 3.8 seems to have stalled as well.

So, as I definitively know how daunting FOSS work is in reality from my own activities in this area, I humbly ask that you don't take any offense on my question. It is just about finding out what your plans are on this. Are you planning to pick up the work on this again?

Keep up the spirit and with kind regards,
Andreas.

[amotl]

From looking around, @confluentinc's @mhowlett added confluentinc/confluent-kafka-python#716 the other day, which resolved confluentinc/confluent-kafka-python#185 by adding an asyncio-based producer example asyncio_example.py.

He also complemented the example by a respective blog post at https://www.confluent.io/blog/kafka-python-asyncio-integration/. Would that be the way to go?

[tvoinarovskyi]

Good day Andreas,
First of all, a big thank you for taking the interest in the state of the library and taking your time to write the inquiry. That is something I very much appreciate and helps me keep motivated. Sadly I had to take a break from open source development, but I start picking up right now.

We do plan to keep supporting the library with regards to bug fixes and compatibility support. In the last few years Kafka Broker went quite far ahead and keeping up is hard, so I don't think I personally will have enough time for all major feature support, such as KIP-227 (Incremental Fetcher) or other KIP that involve major code refactoring. On the other hand, KIPs involving different platform support (Cloud implementations), authorization issues, and not complex improvements (like new compressions support) is something I would like to invest my time in.

A few things I would like to see happening by the end of this year would be:

• Removal of warnings in Python3.8 and Python3.9. It's something I started in fixpy38 branch. The library should still work in 3.8, but the warnings are really bothering me.
• Add typing and black formatting of the code.
• Resolving issues related to cloud providers. I personally did not yet look into those (we only test on released binaries of Kafka).
• Adding Admin functionality if possible, really sorry about leaving @gabriel-tincu hanging with that PR.
• Bugfixes and code review.

As for #629, I will release it ASAP, not sure how I missed it. It's not super critical, as the leak is quite minor and I don't think it will affect a lot of cases.

Best regards,
Taras Voinarovskyi

[tvoinarovskyi]

@amotl About your second question, regarding using confluent-kafka-python. confluent-kafka-python is a stable driver on top of the C library librdkafka, which is fast and has commercial support, so it will have a bit more features. If it works for your cases - no need to avoid using it.

As for asyncio support - the example provided in the blog is very limited, they just run the producer in a background thread. It's not as simple to wrap some functions of Consumer that way, you will need to create a queue to batch or make sure no documents are lost during the process by getting 1 message at a time. It will work fine if the use case is simple static consumption.

For example:

• You get 1 message at a time from a consumer that runs in the background thread. You wait as the example producer showed and get 1 message.
• While you process the message the consumer performs a rebalance and the partition of the message is assigned to another consumer
• You commit the message and get an error, that a rebalance moved the message to a different partition, how should we proceed?

In blocking applications, this does not really happen as rebalancing (at least in Java client, not so sure in librdkafka) does not happen between poll calls. So you can simply process the message and be sure it should just commit (rebalance will wait till you are finished with a timeout). There are ways to provide callbacks for all those processes, but combining those with asyncio will be a good deal of effort. In aiokafka you would still need to provide proper callbacks, but you would not need to solve threading issues in addition to flow control logic.

But this scenario is rather complex, so consider what works for you.

[amotl]

Dear Taras,

thanks a bunch for your detailed elaboration on this topic!

With kind regards,
Andreas.

[bitphage]

Any updates? I see not much progress in PRs review process unfortunately.

[tvoinarovskyi]

Yes, no progress in some time now, sadly. Long story short, changed my job and (with Covid not helping) can't get any spare time to work on opensource. Not sure when I will be able to... If anyone is interested in helping with code review, would gladly accept the help.

[patkivikram]

Do we have any more committers on this project or can this be considered a dead project now that @tvoinarovskyi does not have the bandwidth?

[ods]

Do we have any more committers on this project or can this be considered a dead project now that @tvoinarovskyi does not have the bandwidth?

Do you mean ones having rights to commit? or competent enough to lead the project? The former is not a problem.

[patkivikram]

so that means we can have a continuous release stream correct?

[ods]

Right. I have commit/merge rights and either @tvoinarovskyi or @asvetlov is hopefully still responsive to approve releases. But we need somebody competent and motivated enough to dive into current issues/PRs.

[pawelrubin]

Hi folks ✋🏻

Looking at the name of this issue and the fact that it's still open, I've decided to ask here.

I'm curious if there is anybody currently working on typings for the aiokafka library. If not, I'd more than happy to start working on them, as I need them anyway in my own project.

Btw, I see that black formatting is included in the list provided by @tvoinarovskyi. Could it be introduced? I believe it would significantly increase the readability of the codebase.

I've started with a small #752 that pyupgrades the code.

[ods]

I'm curious if there is anybody currently working on typings for the aiokafka library.

I see that black formatting is included in the list provided by @tvoinarovskyi. Could it be introduced?

Such massive changes invalidate all pull requests that are stale in review. It'd be quite unfriendly step towards their authors.

[pawelrubin]

Such massive changes invalidate all pull requests that are stale in review. It'd be quite unfriendly step towards their authors.

It would only require running black . on the pull request's branch.
I think all authors of the currently opened pull requests would benefit from standardized formatting of the codebase.

I'm curious if there is anybody currently working on typings for the aiokafka library.

is quoted in your comment. Do you have some thoughts on the matter?