bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

1st1 · 2019-05-21T20:49:53Z

This makes it easy to play with asyncio APIs with simply
using async/await in the REPL.

@asvetlov would you mind playing with this? Pull the branch, make sure to rebuild Python, and then ./python -m asyncio. I quite like this and I think it would be a valuable addition to asyncio 3.8.

https://bugs.python.org/issue37028

eamanu

I test it (debian 9) and this is great!

asvetlov · 2019-05-22T08:50:44Z

Looks good, works fine!

I think unittests are needed.
./Lib/test_code_module.py can be used as an example of interactive console testing.

Carreau · 2019-05-22T14:20:42Z

$ python -m asyncio
asyncio REPL

Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import itertools
>>> import asyncio
>>> async def ticker():
...     for i in itertools.count():
...         print(i)
...         await asyncio.sleep(1)
...
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-1' coro=<ticker() running at <console>:1>>
>>> await asyncio.sleep(5)
# no output

You should be able to keep the same eventloop running right ?
Drawback : that would prevent people from running asyncio.run() from within this REPL.

Carreau · 2019-05-22T14:22:48Z

Lib/asyncio/__main__.py

+
+
+if __name__ == '__main__':
+    console = AsyncIOInteractiveConsole(locals())


is the locals() here on purpose ? Seem to leak some values...

$ python -m asyncio asyncio REPL Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43) [Clang 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> code <module 'code' from '/Users/bussonniermatthias/dev/cpython/Lib/code.py'> >>> inspect <module 'inspect' from '/Users/bussonniermatthias/dev/cpython/Lib/inspect.py'> >>> sys <module 'sys' (built-in)> >>> console <__main__.AsyncIOInteractiveConsole object at 0x109ab5be0> >>> banner 'asyncio REPL\n\nPython 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43) \n[Clang 9.0.0 (clang-900.0.39.2)] on darwin\nType "help", "copyright", "credits" or "license" for more information.\n' >>>

Carreau · 2019-05-22T14:38:09Z

And to extend on my comment, here is what I am suggesting as a behavior (but maybe we do that after beta1), where the background coroutines run when there are foreground ones:

>>> import asyncio
>>> import itertools
>>> async def ticker():
...    for i in itertools.count():
...        print(i)
...        await asyncio.sleep(1)
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-2' coro=<ticker() running at <empty co_filename>:1>>
>>> await asyncio.sleep(5)
0
1
2
3
4

The drawback would be :

>>> asyncio.run(ticker())
...
Traceback (most recent call last):
  File "minirepl.py", line 31, in <module>
    asyncio.run(repl())
  [...snip]
  File "", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

Carreau · 2019-05-22T14:42:22Z

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

1st1 · 2019-05-22T21:35:54Z

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

Ah, this is quite interesting. The loop is indeed not running in the background which might confuse some users.

@asvetlov Andrew what do you think? Should we run asyncio event loop in the main thread, and implement the REPL in another non-main thread?

asvetlov · 2019-05-22T22:09:16Z

No, REPL in non-main thread is a bad idea.
At least this solution is very different from my expectations: when I start an interactive session my thread is the main.

njsmith · 2019-05-22T22:39:21Z

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience. Also consider:

>>> async def ticker():
...     while True:
...         await asyncio.sleep(1)
...         print("tick!")
>>> asyncio.create_task(ticker)
>>>

(Or more realistically, something like starting a server in a background task from the REPL.)

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

njsmith · 2019-05-22T22:39:53Z

Oh hah, sorry I skimmed the thread and missed that @Carreau used that exact example already :-)

njsmith · 2019-05-22T22:42:39Z

And the other general approach would be to write the whole REPL async, but that requires support for async stdio, which is complicated (see also).

Carreau · 2019-05-22T23:13:02Z

REPL not in main thread is bad from experience. Many library expect to be in main.

There are two things in play here:
Starting/stoping the loop between each input:

def run():
      input()
      co = compile()
      asyncio.run_untin_complete(co)

Or blocking the loop when requesting input:

async def run():
       while True:
           input()
           co = compile()
           await Function(co)
asyncio.run(run())

I believe they have similar– but not quite identical behavior.
I think that blocking the loop between inputs is more reasonable, as at least you can await sleep(...) to get the bg tasks moving. Which current implementation does not allow.

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well, it works already with IPython :-P

1st1 · 2019-05-23T17:55:12Z

@njsmith

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience.

Agree. I was super confused why @Carreau's examples with while True: print didn't work "as expected". It's important that less obvious examples like "starting a TCP server" would work, i.e. the server should actually accept connections even if there's no user input.

@asvetlov

No, REPL in non-main thread is a bad idea.

I assume you meant "event loop in non-main thread is a bad idea"?

@Carreau

REPL not in main thread is bad from experience. Many library expect to be in main.

Why? What libraries have to do with this?

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well

I'm OK to iterate. If there are open issues we won't push this in 3.8. However if there's no actual iteration on design, I prefer to push things and let people use them. asyncio REPL isn't critical or dependable piece of software, we can fix/tweak it.

As for IPython -- I'm glad that it exists, but I still can't get accustomed to it and always want a quick built-in asyncio repl to test things.

@ALL: I've pushed an updated implementation that runs the REPL in a separate thread, keeping an asyncio event loop always running in the main thread. Seems to work just fine, please try it out.

Lib/asyncio/__main__.py

asvetlov · 2019-05-23T18:28:49Z

Some feedback:

>>> import asyncio

>>> async def f():
...     while(True):
...         print('.')
...         await asyncio.sleep(1)
... 
>>> await f()
.
.
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 434, in result
    raise CancelledError()
concurrent.futures._base.CancelledError

Traceback for keyboard interrupt is confusing a little.
Should I see it if press Ctrl+C to break long-running await func()?

>>> def g():
...     while True:
...         pass
... 
>>> g()
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "<console>", line 3, in g
KeyboardInterrupt

The same question. Maybe REPL can hide it's own part of stack traceback?

The worst part:

>>> t = asyncio.create_task(f())
.
>>> .
.
asd.
sdsd.
s.
d.
.
.
.
sds.
.
[1]    30723 quit (core dumped)  ./python -m asyncio

I've started a task for f() function from the first snippet.

First, dots printed by the task are mixed with my typing. It makes editing REPL string almost impossible.

Also, pressing Ctrl+C when editing does nothing.
In the standard Python REPL it stops editing, prints KeyboardInterrupt without traceback and opens a fresh edit line.

In general, not bad at all if mentioned inconveniences can be fixed.

asvetlov · 2019-05-23T18:33:44Z

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

1st1 · 2019-05-23T18:34:56Z

Some feedback:

Thanks for trying it out :)

Traceback for keyboard interrupt is confusing a little.

Both ^C outputs can be fixed.

The worst part:

This is tricky. I guess the only answer I have to this is "don't print in background tasks".

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is OK for the standard repl, I assume we can dismiss this usability annoyance for the asyncio one.

1st1 · 2019-05-23T18:37:37Z

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

asvetlov · 2019-05-23T18:38:21Z

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is good for the standard repl, I assume we can dismiss this for the asyncio one.

Agree.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Agree again.

asvetlov · 2019-05-23T18:46:58Z

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
>>> 
exiting asyncio REPL...
sys:1: RuntimeWarning: coroutine 'f' was never awaited

Please take a look on the last line. It is printed after the REPL exit.

1st1 · 2019-05-23T18:50:23Z

Please take a look on the last line. It is printed after the REPL exit.

What's wrong with it? The coroutine wasn't actually awaited. I don't think that silencing this warning in the REPL is a good idea. Here's an example interaction where having this warning will hint to the user that they should probably add an "await":

>>> async def foo(): print('hello')
>>> foo()
<coroutine ...>
>>> foo()
<coroutine ...>
sys:1: RuntimeWarning: coroutine 'foo' was never awaited

One other possibility is to filter this warning out while the REPL is exiting...

asvetlov · 2019-05-23T18:53:25Z

I mean it is printed after exiting asyncio REPL... message.
I would like to see the print before the exit message. Pretty sure it requires a very small change :)

asvetlov · 2019-05-23T18:54:27Z

I love that debug mode just works!

andrew•~/projects/cpython(1st1-arepl⚡)» ./python -X dev -m asyncio     
>>> import asyncio

>>> import time
>>> async def f():
...     time.sleep(1)
... 
>>> await f()
Executing <Task finished name='Task-1' coro=<<module>() done, defined at <console>:1> result=None created at /home/andrew/projects/cpython/Lib/asyncio/__main__.py:39> took 1.002 seconds

This makes it easy to play with asyncio APIs with simply using async/await in the REPL.

1st1 · 2019-05-23T19:53:28Z

@asvetlov The latest update should address your comments.

Carreau · 2019-05-23T20:13:47Z

Why? What libraries have to do with this?

from a top of my head i've seem people using open-cv which seemed to not like bing imported in not-main-thread. I'll see if I can your new version to crash :-)

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Well, it is kinda-half-possible; IPython does it; if there are no asyncio task running, and no top-level await we just use a dumb-coroutine iterator. But that's a detail; I agree it's overkill to try-to get both to work.

asvetlov · 2019-05-23T20:31:44Z

@Carreau seems like you fall into the same trap as me.
REPL is for Read–Eval–Print Loop
The current version of PR does Read part in a thread but Eval and Print is executed inside the main thread.
So everything that you are typing is executed in the main thread context.

@njsmith mentioned this way:

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

Looks like "wonky" approach works pretty well.

Carreau · 2019-05-23T20:36:27Z

pasting multiple lines misprints the ...:

$ python -m asyncio
asyncio REPL 3.8.0a4+ (remotes/origin/pr/13472:7a973fb6ec, May 23 2019, 13:31:27)
[Clang 4.0.1 (tags/RELEASE_401/final)] on darwin

Use "await" directly instead of asyncio.run().
Type "help", "copyright", "credits" or "license" for more information.

>>> import asyncio

>>> async def ticker():
    while True:
        await asyncio.sleep(1)
        print("tick!")... ... ...

Carreau · 2019-05-23T21:15:08Z

Nevermind, this is my system that have an issue with readline.

1st1 · 2019-05-23T22:55:07Z

@asvetlov Please feel free to review this. If you green light it I'll merge and add tests later.

Lib/asyncio/__main__.py

tirkarthi · 2019-05-24T08:13:33Z

Misc/NEWS.d/next/Library/2019-05-23-18-57-34.bpo-37028.Vse6Pj.rst

@@ -0,0 +1 @@
+Implement asyncio REPL


Can a note be added to Whats New too?

asvetlov · 2019-05-24T13:27:48Z

Much better but the behavior of Ctrl+C when editing is still different from the standard Python REPL

asvetlov

Anyway, looks good to me

Carreau

This looks good !

Carreau · 2019-05-25T04:33:06Z

We likely can also have entries in tutorials, and pydoc.py for later.

…-13472) This makes it easy to play with asyncio APIs with simply using async/await in the REPL.

1st1 requested a review from asvetlov as a code owner May 21, 2019 20:49

the-knights-who-say-ni added the CLA signed label May 21, 2019

bedevere-bot added the awaiting core review label May 21, 2019

eamanu reviewed May 22, 2019

View reviewed changes

Carreau reviewed May 22, 2019

View reviewed changes

asvetlov reviewed May 23, 2019

View reviewed changes

Lib/asyncio/__main__.py Show resolved Hide resolved

1st1 added 3 commits May 23, 2019 15:08

Implement asyncio REPL (activated via 'python -m asyncio')

17e2c05

This makes it easy to play with asyncio APIs with simply using async/await in the REPL.

Run REPL in a thread; keep asyncio loop always busy in the main thread

dc35401

Better handle ^C; filter pointless warnings on exit; edit banner

7a973fb

1st1 force-pushed the arepl branch from 81ddf88 to 7a973fb Compare May 23, 2019 19:53

Add NEWS file

3264679

1st1 changed the title ~~Implement asyncio REPL (activated via 'python -m asyncio')~~ bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') May 23, 2019

nit-fixes

00eb003

Carreau reviewed May 23, 2019

View reviewed changes

Lib/asyncio/__main__.py Outdated Show resolved Hide resolved

Cleanup newlines; make input() thread daemon

284a8b1

tirkarthi reviewed May 24, 2019

View reviewed changes

asvetlov approved these changes May 24, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 24, 2019

Carreau approved these changes May 25, 2019

View reviewed changes

1st1 merged commit 16cefb0 into python:master May 27, 2019

bedevere-bot removed the awaiting merge label May 27, 2019

1st1 deleted the arepl branch May 27, 2019 11:42

brandtbucher mentioned this pull request Jun 11, 2019

python -m static_frame static-frame/static-frame#61

Merged

darthdeus mentioned this pull request Nov 24, 2019

await does not work in ipdb, even though IPython supports it gotcha/ipdb#174

Closed

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-37028: asyncio REPL; activated via 'python -m asyncio'. (pythonGH…

54debeb

…-13472) This makes it easy to play with asyncio APIs with simply using async/await in the REPL.

weiji14 mentioned this pull request Jul 31, 2023

✨ AsyncIterableWrapper for creating asynchronous iterators weiji14/bambooflow#6

Merged

3 tasks

1st1 mentioned this pull request Oct 17, 2023

Implement asyncio repl #81209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

1st1 commented May 21, 2019 •

edited by bedevere-bot

Loading

eamanu left a comment

asvetlov commented May 22, 2019

Carreau commented May 22, 2019

Carreau May 22, 2019

Carreau commented May 22, 2019

Carreau commented May 22, 2019

1st1 commented May 22, 2019

asvetlov commented May 22, 2019

njsmith commented May 22, 2019

njsmith commented May 22, 2019

njsmith commented May 22, 2019 •

edited

Loading

Carreau commented May 22, 2019

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019 •

edited

Loading

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019

Carreau commented May 23, 2019

asvetlov commented May 23, 2019

Carreau commented May 23, 2019

Carreau commented May 23, 2019

1st1 commented May 23, 2019

tirkarthi May 24, 2019

asvetlov commented May 24, 2019

asvetlov left a comment

Carreau left a comment

Carreau commented May 25, 2019



		if __name__ == '__main__':
		console = AsyncIOInteractiveConsole(locals())

bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

Conversation

1st1 commented May 21, 2019 • edited by bedevere-bot Loading

eamanu left a comment

Choose a reason for hiding this comment

asvetlov commented May 22, 2019

Carreau commented May 22, 2019

Carreau May 22, 2019

Choose a reason for hiding this comment

Carreau commented May 22, 2019

Carreau commented May 22, 2019

1st1 commented May 22, 2019

asvetlov commented May 22, 2019

njsmith commented May 22, 2019

njsmith commented May 22, 2019

njsmith commented May 22, 2019 • edited Loading

Carreau commented May 22, 2019

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019 • edited Loading

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019

asvetlov commented May 23, 2019

asvetlov commented May 23, 2019

1st1 commented May 23, 2019

Carreau commented May 23, 2019

asvetlov commented May 23, 2019

Carreau commented May 23, 2019

Carreau commented May 23, 2019

1st1 commented May 23, 2019

tirkarthi May 24, 2019

Choose a reason for hiding this comment

asvetlov commented May 24, 2019

asvetlov left a comment

Choose a reason for hiding this comment

Carreau left a comment

Choose a reason for hiding this comment

Carreau commented May 25, 2019

1st1 commented May 21, 2019 •

edited by bedevere-bot

Loading

njsmith commented May 22, 2019 •

edited

Loading

1st1 commented May 23, 2019 •

edited

Loading