0.15.0+ development priorities

[x42005e1f]

Greetings to everyone reading this. It has been 8 months since the last release. In a sense, this is quite a long time, and a lot has been done during this period: the number of commits has more than doubled, and the list of recent changes is now so long that it takes up more than half of the entire page on Read the Docs. However, the goal of 0.15.0, that is, upgrading to the beta development stage, still remains unachieved.

By upgrading to the beta development stage in the context of this project, I mean the following:

1. Test coverage, with each of the supported libraries. To this end, I started writing a testing framework — aiologic.testing — but it requires further development (even in architectural terms) and does not include stress tests. In fact, aiologic remains quite reliable even without automated tests, as I regularly check it myself, both theoretically and practically, but this does not protect against regressions and various kinds of silly mistakes.
2. Documentation coverage, both automatically (via sphinx.ext.autodoc) and manually. This is actually more complicated than it seems, and just to support typing on the Sphinx side, a lot of effort had to be put into changing the entire code (since sphinx.ext.autodoc does not support stub files for pure Python modules, and existing third-party extensions just do not work). At the moment, only one section is well written — "Libraries" — and even that does not cover all questions: in particular, it should also include explanations about monkey patching support (quick answer: aiologic supports it, but the required dependencies do not, so import aiologic before monkey patching) and support for native threads (quick answer: most of aiologic's features are independent of thread type, but for some features you may need to import the greenlet module). The situation has improved somewhat with DeepWiki, but it lies a lot and cannot explain what it does not know itself.
3. Confidence in the reliability and minimal surprises of the library's operation. This is perhaps the key difference in how the beta development stage is understood here and how it is understood in the wild (in fact, it is very similar to the common understanding of the stable development stage, but still allows incompatible changes without increasing the major version — the equivalent of this in alpha and beta here is increasing the minor version). Most of the effort has been made in this direction, and one of the latest has been a significant improvement in working with timeouts, and more will be done.

So, it is becoming clear that 0.15.0 may not be released even before the new year. This is influenced by many factors, including those beyond my control, and as the only active developer, I can only do what is within my power (even though I do have some kind of superpowers). The reason why I am not releasing an update until the main goal is achieved, as yet another alpha release, is that for all the public use cases I have seen, 0.14.0 is sufficient (which is usually a fairly simple use of aiologic.Event/aiologic.Lock and threading + asyncio). For cases where you need more features and relevant fixes, you can use the latest version (from GitHub), but I recommend updating at least one hour after the last commit (because during that time I may find some mistakes and quickly fix them via force-push, since separate commits are redundant for this, and there is no one else to do reviews besides me). 0.15.0 is out (and it is still alpha)!

As for the popularity of the project, I will be honest and say that I actively use GitHub and Read the Docs tools to get traffic statistics, as well as search engines to find at least some mentions. Currently, based on the low repository view statistics, near-zero documentation view statistics, almost complete lack of mentions other than my own, and low number of unique uses (discoverable via GitHub), aiologic looks like an almost unknown and most likely misunderstood project (indeed, the topic is very unusual, and the approaches I use are even more unusual, and are the result of work done years before the first release of aiologic). However, you may find that the package is frequently downloaded according to PyPI statistics, but here you should interpret the results correctly: these statistics reflect any attempt to install or download the package, regardless of the package manager used, regardless of how (whether manually, automatically via CI/CD, or as a dependency of another package). Apache Airflow made the main contribution to the PyPI statistics, from 3.0.0rc2 to 3.0.3 (not inclusive). This is also reflected in the aiologic's dependents — almost all dependents since the appearance of aiologic as a dependency of apache-airflow-task-sdk are somehow related to Apache Airflow and only list aiologic in requirements.txt or lock files as its dependency. Thus, there is no information about how many people actually use aiologic directly.

Okay, why did I write all this? Since mid-October, I have been recording unusually high activity in viewing documentation (about a dozen per day) combined with strange spikes in the number of clones, up to two thousand in the middle of last week (however, I have been observing such unusual activity with clones for a long time, and I suspect deployment mechanisms are involved, but I do not have any related details). This led me to wonder how useful the library is at the moment and what priorities should be set to make the latest version as accessible as possible here and now. Unfortunately, I cannot do this without feedback, so I ask everyone interested to express their opinion by voting (or separate comment/discussion/whatever else).

It is okay if no one votes — I will just continue to move forward on my intuition.

[fleming79]

Hi,
Thanks for creating this fantastic module. I think it could be highly relevant and popular in the future.

I'm seriously considering adding aiologic as a dependency for the package I created 'async-kernel' (fairly new and largely unknown), you can see the PR here that I'm currently working on.

I'd like to provide some feedback on Event & REvent based on my recent experience:

• I was a bit surprised by how the set/await behaviour appears to be 'preemptive' across threads.
• The __bool__ override was also unexpected behaviour considering that is_set gives the same result.

[fleming79]

Have you considered something like this? (using code from aiologic v0.14.0)

## class Event:


    def __await__(self, /):
        return self.wait_async().__await__()
    
    async def wait_async(self):
        rescheduled = False

        if self.__is_unset:
            self.__waiters.append(event := AsyncEvent())

            if self.__is_unset:
                success = False

                try:
                    success = await event
                finally:
                    if not success:
                        try:
                            self.__waiters.remove(event)
                        except ValueError:
                            pass

                rescheduled = True
            else:
                success = True
        else:
            success = True

        if success:
            self.__wakeup()

        if not rescheduled:
            await checkpoint()

        return success

[x42005e1f]

Have you considered something like this? (using code from aiologic v0.14.0)

Yes, I considered adding related methods, but decided it would be too redundant. Copying code would increase support costs, while delegating would divide methods into "fast" and "not so fast" due to the associated overhead. At the same time, await event looks quite concise (and does not include the word "wait" twice).

It seems to me that this topic is somewhat similar to the problem of using lambda expressions. Defining an explicit function that waits for an event can make code easier to read. In addition, it is easier to extend such a function.

So I do not know what to do about this yet. From an interface design perspective, I need to proceed with caution here.

[x42005e1f]

0.15.0 is out! Now you can use all the latest features right from PyPI.

[x42005e1f]

I take no offense from your comments and would welcome your review of my PR if you feel so inclined. I'm still making some tweaks to the code.

I think you could consider using the low-level events instead of aiologic.Event in cases where there is a single waiter. They are much more lightweight, and their API has stabilized in 0.15.0. Also, your code looks like it could be simplified in some places using aiologic.SimpleQueue or aiologic.Condition (particularly in as_completed()).

Hint: if your code contains event.clear(), then, with rare exceptions, this is a sign of poor design that can be improved by using more appropriate primitives. This is especially important because event.clear() can "eat" signals that were sent during a race condition.

[fleming79]

Thank you for providing the release!
I'll have a look at the optimizations when I get a chance. I'm still just getting to know the capabilities of aiologic.

[fleming79]

There are times where it its helpful and you don't really want to write a wrapper every time.

I guess for that matter I think it could be.

async def wait_async(self):
    if not self:
        await self

[x42005e1f]

I agree, but I have not decided what to do about this yet. And your code does not take checkpoints into account (which are relevant for AnyIO and Trio). Here, it is enough to simply do return await self.

[x42005e1f]

Now there is aiologic.meta.await_for():

from aiologic.meta import await_for

...

tg.start_soon(await_for, event)  # tg is a task group (anyio/trio)