-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[QUESTION] Future development and maintenance of aiokafka #665
Comments
From looking around, @confluentinc's @mhowlett added confluentinc/confluent-kafka-python#716 the other day, which resolved confluentinc/confluent-kafka-python#185 by adding an asyncio-based producer example He also complemented the example by a respective blog post at https://www.confluent.io/blog/kafka-python-asyncio-integration/. Would that be the way to go? |
Good day Andreas, We do plan to keep supporting the library with regards to bug fixes and compatibility support. In the last few years Kafka Broker went quite far ahead and keeping up is hard, so I don't think I personally will have enough time for all major feature support, such as KIP-227 (Incremental Fetcher) or other KIP that involve major code refactoring. On the other hand, KIPs involving different platform support (Cloud implementations), authorization issues, and not complex improvements (like new compressions support) is something I would like to invest my time in. A few things I would like to see happening by the end of this year would be:
As for #629, I will release it ASAP, not sure how I missed it. It's not super critical, as the leak is quite minor and I don't think it will affect a lot of cases. Best regards, |
@amotl About your second question, regarding using confluent-kafka-python. As for asyncio support - the example provided in the blog is very limited, they just run the producer in a background thread. It's not as simple to wrap some functions of Consumer that way, you will need to create a queue to batch or make sure no documents are lost during the process by getting 1 message at a time. It will work fine if the use case is simple static consumption. For example:
In blocking applications, this does not really happen as rebalancing (at least in Java client, not so sure in librdkafka) does not happen between poll calls. So you can simply process the message and be sure it should just commit (rebalance will wait till you are finished with a timeout). There are ways to provide callbacks for all those processes, but combining those with But this scenario is rather complex, so consider what works for you. |
Dear Taras, thanks a bunch for your detailed elaboration on this topic! With kind regards, |
Any updates? I see not much progress in PRs review process unfortunately. |
Yes, no progress in some time now, sadly. Long story short, changed my job and (with Covid not helping) can't get any spare time to work on opensource. Not sure when I will be able to... If anyone is interested in helping with code review, would gladly accept the help. |
Do we have any more committers on this project or can this be considered a dead project now that @tvoinarovskyi does not have the bandwidth? |
Do you mean ones having rights to commit? or competent enough to lead the project? The former is not a problem. |
so that means we can have a continuous release stream correct? |
Right. I have commit/merge rights and either @tvoinarovskyi or @asvetlov is hopefully still responsive to approve releases. But we need somebody competent and motivated enough to dive into current issues/PRs. |
Hi folks ✋🏻 Looking at the name of this issue and the fact that it's still open, I've decided to ask here. I'm curious if there is anybody currently working on typings for the aiokafka library. If not, I'd more than happy to start working on them, as I need them anyway in my own project. Btw, I see that black formatting is included in the list provided by @tvoinarovskyi. Could it be introduced? I believe it would significantly increase the readability of the codebase. I've started with a small #752 that |
Such massive changes invalidate all pull requests that are stale in review. It'd be quite unfriendly step towards their authors. |
It would only require running
is quoted in your comment. Do you have some thoughts on the matter? |
Dear @asvetlov and @tvoinarovskyi,
first things first: Thanks a bunch for conceiving and maintaining aiokafka during the last years. You know who you are.
Now, I am humbly asking how you are looking at the future of this library. I see that #628 has been fixed by @iamsinghrajat the other day within #629 and #659 has been fixed by @dkilgore90 within #660 (thanks for these!). Also, @tvoinarovskyi already started working on Python 3.8 support on behalf of the
fixpy38
branch with 946bbe9.While #629 has been merged already and made it into a
v0.6.1.dev0
pre-release, it did not make it into an official release available on PyPI. On the other hand, #660 is still open and the work on supporting Python 3.8 seems to have stalled as well.So, as I definitively know how daunting FOSS work is in reality from my own activities in this area, I humbly ask that you don't take any offense on my question. It is just about finding out what your plans are on this. Are you planning to pick up the work on this again?
Keep up the spirit and with kind regards,
Andreas.
The text was updated successfully, but these errors were encountered: