Core: Add async support to kivy App

[matham]

This adds async support to kivy. It is a continuation from #5241.

It only adds support for async, without support for the async_bind() convenience method as requested. The event/property binding will (may) come later in a different PR.

PR info copied from the docs:

Background

Normally, when a Kivy app is run, it blocks the thread that runs it until the app exits. Internally, at each clock iteration it executes all the app callbacks, handles graphics and input, and idles by sleeping for any remaining time.

To be able to run asynchronously, the Kivy app may not sleep, but instead must release control of the running context to the asynchronous event loop running the Kivy app. We do this when idling by calling the appropriate functions of the async package being used instead of sleeping.

Async configuration

To run an async app, both the KIVY_EVENTLOOP environmental variable must be set appropriately, and :func:async_runTouchApp or :meth:App.async_run must be scheduled to run in the external async package's event loop. The variable tells kivy which async library to use when idling and
:func:async_runTouchApp or :meth:App.async_run run the actual app.

The environmental variable KIVY_EVENTLOOP determines which async library to use, if at all. It can be set to one of "sync" when it should be run synchronously like a normal app, "async" when the standard library asyncio should be used, or "trio" if the trio library should be used. If not set it
defaults to "sync".

In the "async" or "trio" case, one schedules :func:async_runTouchApp or :meth:App.async_run to run within the given library's async event loop as in the examples shown below. Kivy is then treated as just another coroutine that the given library runs in its event loop.

For a fuller basic and more advanced examples, see the demo apps in examples/async.

Asyncio example

    import asyncio
    import os
    os.environ['KIVY_EVENTLOOP'] = 'async'

    from kivy.app import async_runTouchApp
    from kivy.uix.label import Label


    loop = asyncio.get_event_loop()
    loop.run_until_complete(async_runTouchApp(Label(text='Hello, World!')))
    loop.close()

Trio example

    import trio
    import os
    os.environ['KIVY_EVENTLOOP'] = 'trio'

    from kivy.app import async_runTouchApp
    from kivy.uix.label import Label

    trio.run(async_runTouchApp, Label(text='Hello, World!'))

Interacting with Kivy app from other coroutines

It is fully safe to interact with any kivy object from other coroutines running within the same async event loop. This is because they are all running from the same thread and the other coroutines are only executed when Kivy is idling.

Similarly, the kivy callbacks may safely interact with objects from other coroutines running in the same event loop. Normal single threaded rules apply to both case.

Examples

I added a few full examples to examples/async, and the docs are pretty complete. Please see those examples to get a fuller understanding of how this would be used.

Testing

With this PR, I also added a new way of graphically testing apps. Using pytest, I added a pytest kivy_app fixture that takes a function that returns a Kivy app, and creates a self-contained kivy app from it. With it, we can test apps super easily and they get re-created for each test. I hope this will make it much easier for people to add tests to test complex widgets. I created a UnitKivyApp base class to help with this testing as it provides some convenient methods.

Please see kivy/tests/test_app.py for various example tests and kivy/tests/async_common for the UnitKivyApp base class. I actually am using this to help test a very large app I made, which I'm not sure how I would have tested the GUI without this kind of thing.

One note, these tests are skipped for python 3.5 because support for yielding from a async def method was only added in 3.6 and we need this to help with testing.

Here's a couple of the sample tests to show how easy async make testing actual full kivy apps:

def button_app():
    from kivy.app import App
    from kivy.uix.togglebutton import ToggleButton

    class TestApp(UnitKivyApp, App):
        def build(self):
            return ToggleButton(text='Hello, World!')

    return TestApp()


@async_run(app_cls_func=button_app)
async def test_button_app(kivy_app):
    assert kivy_app.root.text == 'Hello, World!'
    assert kivy_app.root.state == 'normal'

    async for state, touch_pos in kivy_app.do_touch_down_up(
            widget=kivy_app.root, widget_jitter=True):
        pass

    assert kivy_app.root.state == 'down'

And another:

def text_app():
    from kivy.app import App
    from kivy.uix.textinput import TextInput

    class TestApp(UnitKivyApp, App):
        def build(self):
            return TextInput()

    return TestApp()


@async_run(app_cls_func=text_app)
async def test_text_app(kivy_app):
    text = kivy_app.root
    assert text.text == ''

    # activate widget
    async for state, touch_pos in kivy_app.do_touch_down_up(widget=text):
        pass

    async for state, value in kivy_app.do_keyboard_key(key='A', num_press=4):
        pass
    async for state, value in kivy_app.do_keyboard_key(key='q', num_press=3):
        pass

    assert text.text == 'AAAAqqq'

[HeyITGuyFixIt]

How is this coming? Any progress?

[matham]

It works nicely, but I haven't found the time yet to add the docs and tests.

[matham]

I believe the PR is complete. I updated the first comment with more details.

The osx tests are failing for reasons unrelated to this PR.

[MikahB]

Anxiously awaiting this - have a project with existing substantial async Python3.7 code base where customer now wants to add a touch screen. Kivy + async will be awesome for this!

[matham]

I plan on merging in a week if no feedback or objections is forthcoming. But you should go ahead and try this branch to see if you run into issues and things need to be fixed.

Although I tested with a fairly complex app and things seemed to work fine.

[MikahB]

I plan on merging in a week if no feedback or objections is forthcoming. But you should go ahead and try this branch to see if you run into issues and things need to be fixed.

Although I tested with a fairly complex app and things seemed to work fine.

I doubt I'll have time to get it rolling in the next week, but will definitely post back if I do. Thanks again for your work on this!

[gottadiveintopython]

Running tests with both pytest-asyncio and pytest-trio installed => success
Running tests with pytest-asyncio installed without pytest-trio installed => success
Running tests with pytest-trio installed without pytest-asyncio installed => fail

        if _is_trio_fixture(fixturedef.func, coerce_async, kwargs):
            if request.scope != "function":
                raise RuntimeError("Trio fixtures must be function-scope")
            if not is_trio_test:
>               raise RuntimeError("Trio fixtures can only be used by Trio tests")
E               RuntimeError: Trio fixtures can only be used by Trio tests

Tests always fail if pytest-asyncio is not installed. Setting KIVY_EVENTLOOP to trio didn't change the tests result. I'm not sure this is an expected behavior or not though.

Versions

• Python 3.7.1
• LinuxMint 18.2 Mate Edition 64bit

[matham]

Thanks for testing!

Hmm, strange. I didn't see this behavior, maybe because I had a pytest.ini. There is a issue if using pytest_trio when pytest_asyncio is installed (hence why you need to do https://pytest-trio.readthedocs.io/en/latest/quickstart.html#enabling-trio-mode-and-running-your-first-async-tests), but I thought I managed to work around this. I reported this issue to njsmith, which he reported here: pytest-dev/pytest-asyncio#124 (comment).

I'll try to reproduce in a clean venv.

[gottadiveintopython]

Ah sorry, it was my mistake 😅 . I forgot about that (trio_mode). After I created pytest.ini:

[pytest]
trio_mode = true

the tests passed.

[akloster]

Hm, I do understand better now. I'm not familiar enough with the way kivy initializes things or why you need to initialize anything about asyncio before creating a loop. It normally really doesn't do anything with sideeffects or blocking unless it is waiting for events on file descriptors. The difference between Windows and Linux for asyncio is mainly which event loop class is initialized and those classes have different ways of blocking for events. Two years ago I integrated asyncio into a kivy application the same way I did it with Blender and IPython, the latter of which now has better facilities for that. I basically run the _run_once member function of the loop from some idle callback. Now, normally that function blocks until the next event occurs, but you can stop it two ways: One is by providing some event to be called soon/1ms later, or subclass the loop class you will be using (which is different depending on the platform) and override the _run_once method with a copy that calls the "select" function with a short timeout. Now I can run Kivy unmolested, I can use all of asyncio and third-party libraries and I can even interact with the loop from kivy events. I have to look some more into the pull request how things work now. Please don't take my comments as criticism or attacks, I'm just trying to give my opinion and understand Kivy better.

…

[gottadiveintopython]

ValueError: A parser named app already exists

This error might have to do with GraphicsUnitTest, because

• python -m pytest ./kivy/tests/test_app.py::AppTest ./kivy/tests/test_app.py::test_basic_app -> fail
• python -m pytest ./kivy/tests/test_app.py::test_basic_app ./kivy/tests/test_app.py::AppTest -> success

(AppTest is a subclass of GraphicsUnitTest.)

I'm not sure GraphicsUnitTest works while KIVY_EVENTLOOP=trio though.

[akloster]

Is the main loop supposed to exit when the app finishes? I am running into the situation that when closing the application by ESC, the loop seems to shut down before other running tasks get a chance to clean up after them.

[akloster]

The problem about async_run closing the loop was caused by "run_until_complete". "run_forever" works better, if another coroutine stops the loop later.

Next issue: I'm trying to set widget properties (like label texts) and that seems to break something. At the very least the changed properties don't show up on the screen.

[gottadiveintopython]

@akloster

The problem about async_run...

Hi, it's just an example. Kivy itself doesn't force you to use run_until_complete(), doesn't it?

Next issue...

Can you show me the code that reproduce that issue?

[akloster]

I had these issues in an app that uses aiohttp's Websocket client to read Flightgear simulation data. So this wouldn't be easy to share/reproduce. Basically aiohttp complained that the websocket/client session was still running when the loop had been shut down.

Originally I tried to schedule coroutines from kivy event handlers, and that didn't work at all. Now it works, so it's possible to use on_start and on_stop to start and stop background service tasks.

An example I can share is this:
https://gist.github.com/akloster/a5446416cf9a99f208797e5fa289e53c

This example starts a main task from inside the App instance and stops it from on_stop. Catching CancelledError, the main' task simulates an asynchronous cleanup through asyncio.sleep`.

That sleep task won't yield while the app is running, so I run the loop again with that task. Two disadvantages: The main block of the Python program has to know about that task. Secondly, any exceptions from the main task are only presented after the app closes.

I'll probably figure out another way. It's probably just me not understanding the complete picture...

[gottadiveintopython]

The main block of the Python program has to know about that task

So you want to start main_task() when on_start is triggered, and want to stop it when the app is closed, but you want the app to be not aware of main_task()? You mean like this? I'm not familiar with asyncio so there might be a better way though.

[akloster]

@gottadiveintopython yes, there are multiple solutions. I settled on overriding "async_run" in a custom Application.

[gottadiveintopython]

@akloster ah ok, so the issues you had were already solved? I thought you currently have some issues.

[matham]

I've publicly added the init_async_lib method, which can be called to set the async library. But also added the async_lib param to async_run so that you don't have to use the environment variable or even init_async_lib directly, like @akloster requested.

But the environment variable still works like before, it's just now optional. I also updated the examples and docs for this.

I'm pretty happy with this now, and would merge maybe next weekend if there's no additional feedback.

[gottadiveintopython]

I've just run tests. That error (ValueError: A parser named app already exists) doesn't occur anymore. nice.

[intgsull]

This is awesome as I was just looking for a way to integrate asyncio and kivy. I apologize for the ignorance but how can I use this PR as it's not included in the latest stable release?

[ysangkok]

@intgsull I think you can do pip install https://github.com/kivy/kivy/archive/master.zip

EDIT: better just follow the complete guide https://kivy.org/doc/stable/installation/installation-devel.html#installation-devel

[matham]

Also, look in the install docs and depending on the platform there may be a nightly wheel or daily PPA to try.

[pidou46]

Big thanks for this major PR.

By the way there is a minor issue with the Asyncio example given on this page:
os.environ['KIVY_EVENTLOOP'] = 'async'
Should be:
os.environ['KIVY_EVENTLOOP'] = 'asyncio'

[goodboy]

@matham I'm just porting to mainline (since I just found out your old branch was put to rest) and I'm not sure how to deal with the lack of AsyncBindQueue and .async_bind() that's missing in this PR. Ideally I can still iterate event streams async and not have to rejig my app. I'm wondering what's the recommended approach to take now for this?

[mihow]

@matham Thank you for this great work. Can you provide an example of how to call an async method without the .async_bind() convenience method? For example, how can I make a button widget start a long running background task? Can I use asyncio.create_task() within a Widget class?

[mihow]

@goodboy Did you find an alternative solution to .async_bind()?