Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') #13472

Merged
merged 6 commits into from May 27, 2019

Conversation

Projects
None yet
8 participants
@1st1
Copy link
Member

commented May 21, 2019

This makes it easy to play with asyncio APIs with simply
using async/await in the REPL.

@asvetlov would you mind playing with this? Pull the branch, make sure to rebuild Python, and then ./python -m asyncio. I quite like this and I think it would be a valuable addition to asyncio 3.8.

https://bugs.python.org/issue37028

@eamanu
Copy link
Contributor

left a comment

I test it (debian 9) and this is great!

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

Looks good, works fine!

I think unittests are needed.
./Lib/test_code_module.py can be used as an example of interactive console testing.

@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

$ python -m asyncio
asyncio REPL

Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import itertools
>>> import asyncio
>>> async def ticker():
...     for i in itertools.count():
...         print(i)
...         await asyncio.sleep(1)
...
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-1' coro=<ticker() running at <console>:1>>
>>> await asyncio.sleep(5)
# no output

You should be able to keep the same eventloop running right ?
Drawback : that would prevent people from running asyncio.run() from within this REPL.



if __name__ == '__main__':
console = AsyncIOInteractiveConsole(locals())

This comment has been minimized.

Copy link
@Carreau

Carreau May 22, 2019

Contributor

is the locals() here on purpose ? Seem to leak some values...

$ python -m asyncio
asyncio REPL

Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> code
<module 'code' from '/Users/bussonniermatthias/dev/cpython/Lib/code.py'>
>>> inspect
<module 'inspect' from '/Users/bussonniermatthias/dev/cpython/Lib/inspect.py'>
>>> sys
<module 'sys' (built-in)>
>>> console
<__main__.AsyncIOInteractiveConsole object at 0x109ab5be0>
>>> banner
'asyncio REPL\n\nPython 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43) \n[Clang 9.0.0 (clang-900.0.39.2)] on darwin\nType "help", "copyright", "credits" or "license" for more information.\n'
>>>
@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

And to extend on my comment, here is what I am suggesting as a behavior (but maybe we do that after beta1), where the background coroutines run when there are foreground ones:

>>> import asyncio
>>> import itertools
>>> async def ticker():
...    for i in itertools.count():
...        print(i)
...        await asyncio.sleep(1)
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-2' coro=<ticker() running at <empty co_filename>:1>>
>>> await asyncio.sleep(5)
0
1
2
3
4

The drawback would be :

>>> asyncio.run(ticker())
...
Traceback (most recent call last):
  File "minirepl.py", line 31, in <module>
    asyncio.run(repl())
  [...snip]
  File "", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

Ah, this is quite interesting. The loop is indeed not running in the background which might confuse some users.

@asvetlov Andrew what do you think? Should we run asyncio event loop in the main thread, and implement the REPL in another non-main thread?

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

No, REPL in non-main thread is a bad idea.
At least this solution is very different from my expectations: when I start an interactive session my thread is the main.

@njsmith

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience. Also consider:

>>> async def ticker():
...     while True:
...         await asyncio.sleep(1)
...         print("tick!")
>>> asyncio.create_task(ticker)
>>> 

(Or more realistically, something like starting a server in a background task from the REPL.)

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

@njsmith

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

Oh hah, sorry I skimmed the thread and missed that @Carreau used that exact example already :-)

@njsmith

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

And the other general approach would be to write the whole REPL async, but that requires support for async stdio, which is complicated (see also).

@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

REPL not in main thread is bad from experience. Many library expect to be in main.

There are two things in play here:
Starting/stoping the loop between each input:

def run():
      input()
      co = compile()
      asyncio.run_untin_complete(co)

Or blocking the loop when requesting input:

async def run():
       while True:
           input()
           co = compile()
           await Function(co)
asyncio.run(run())

I believe they have similar– but not quite identical behavior.
I think that blocking the loop between inputs is more reasonable, as at least you can await sleep(...) to get the bg tasks moving. Which current implementation does not allow.

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well, it works already with IPython :-P

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

@njsmith

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience.

Agree. I was super confused why @Carreau's examples with while True: print didn't work "as expected". It's important that less obvious examples like "starting a TCP server" would work, i.e. the server should actually accept connections even if there's no user input.

@asvetlov

No, REPL in non-main thread is a bad idea.

I assume you meant "event loop in non-main thread is a bad idea"?

@Carreau

REPL not in main thread is bad from experience. Many library expect to be in main.

Why? What libraries have to do with this?

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well

I'm OK to iterate. If there are open issues we won't push this in 3.8. However if there's no actual iteration on design, I prefer to push things and let people use them. asyncio REPL isn't critical or dependable piece of software, we can fix/tweak it.

As for IPython -- I'm glad that it exists, but I still can't get accustomed to it and always want a quick built-in asyncio repl to test things.


@ALL: I've pushed an updated implementation that runs the REPL in a separate thread, keeping an asyncio event loop always running in the main thread. Seems to work just fine, please try it out.

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

Some feedback:

>>> import asyncio

>>> async def f():
...     while(True):
...         print('.')
...         await asyncio.sleep(1)
... 
>>> await f()
.
.
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 434, in result
    raise CancelledError()
concurrent.futures._base.CancelledError

Traceback for keyboard interrupt is confusing a little.
Should I see it if press Ctrl+C to break long-running await func()?

>>> def g():
...     while True:
...         pass
... 
>>> g()
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "<console>", line 3, in g
KeyboardInterrupt

The same question. Maybe REPL can hide it's own part of stack traceback?

The worst part:

>>> t = asyncio.create_task(f())
.
>>> .
.
asd.
sdsd.
s.
d.
.
.
.
sds.
.
[1]    30723 quit (core dumped)  ./python -m asyncio

I've started a task for f() function from the first snippet.

First, dots printed by the task are mixed with my typing. It makes editing REPL string almost impossible.

Also, pressing Ctrl+C when editing does nothing.
In the standard Python REPL it stops editing, prints KeyboardInterrupt without traceback and opens a fresh edit line.

In general, not bad at all if mentioned inconveniences can be fixed.

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

Some feedback:

Thanks for trying it out :)

Traceback for keyboard interrupt is confusing a little.

Both ^C outputs can be fixed.

The worst part:

This is tricky. I guess the only answer I have to this is "don't print in background tasks".

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is OK for the standard repl, I assume we can dismiss this usability annoyance for the asyncio one.

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is good for the standard repl, I assume we can dismiss this for the asyncio one.

Agree.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Agree again.

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
>>> 
exiting asyncio REPL...
sys:1: RuntimeWarning: coroutine 'f' was never awaited

Please take a look on the last line. It is printed after the REPL exit.

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

Please take a look on the last line. It is printed after the REPL exit.

What's wrong with it? The coroutine wasn't actually awaited. I don't think that silencing this warning in the REPL is a good idea. Here's an example interaction where having this warning will hint to the user that they should probably add an "await":

>>> async def foo(): print('hello')
>>> foo()
<coroutine ...>
>>> foo()
<coroutine ...>
sys:1: RuntimeWarning: coroutine 'foo' was never awaited

One other possibility is to filter this warning out while the REPL is exiting...

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

I mean it is printed after exiting asyncio REPL... message.
I would like to see the print before the exit message. Pretty sure it requires a very small change :)

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

I love that debug mode just works!

andrew•~/projects/cpython(1st1-arepl⚡)» ./python -X dev -m asyncio     
>>> import asyncio

>>> import time
>>> async def f():
...     time.sleep(1)
... 
>>> await f()
Executing <Task finished name='Task-1' coro=<<module>() done, defined at <console>:1> result=None created at /home/andrew/projects/cpython/Lib/asyncio/__main__.py:39> took 1.002 seconds

@1st1 1st1 force-pushed the 1st1:arepl branch from 81ddf88 to 7a973fb May 23, 2019

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

@asvetlov The latest update should address your comments.

@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

Why? What libraries have to do with this?

from a top of my head i've seem people using open-cv which seemed to not like bing imported in not-main-thread. I'll see if I can your new version to crash :-)

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Well, it is kinda-half-possible; IPython does it; if there are no asyncio task running, and no top-level await we just use a dumb-coroutine iterator. But that's a detail; I agree it's overkill to try-to get both to work.

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

@Carreau seems like you fall into the same trap as me.
REPL is for Read–Eval–Print Loop
The current version of PR does Read part in a thread but Eval and Print is executed inside the main thread.
So everything that you are typing is executed in the main thread context.

@njsmith mentioned this way:

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

Looks like "wonky" approach works pretty well.

@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

pasting multiple lines misprints the ...:

$ python -m asyncio
asyncio REPL 3.8.0a4+ (remotes/origin/pr/13472:7a973fb6ec, May 23 2019, 13:31:27)
[Clang 4.0.1 (tags/RELEASE_401/final)] on darwin

Use "await" directly instead of asyncio.run().
Type "help", "copyright", "credits" or "license" for more information.

>>> import asyncio

>>> async def ticker():
    while True:
        await asyncio.sleep(1)
        print("tick!")... ... ...
@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

Nevermind, this is my system that have an issue with readline.

@1st1

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

@asvetlov Please feel free to review this. If you green light it I'll merge and add tests later.

@1st1 1st1 changed the title Implement asyncio REPL (activated via 'python -m asyncio') bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') May 23, 2019

Show resolved Hide resolved Lib/asyncio/__main__.py Outdated
@@ -0,0 +1 @@
Implement asyncio REPL

This comment has been minimized.

Copy link
@tirkarthi

tirkarthi May 24, 2019

Contributor

Can a note be added to Whats New too?

@asvetlov

This comment has been minimized.

Copy link
Contributor

commented May 24, 2019

Much better but the behavior of Ctrl+C when editing is still different from the standard Python REPL

@asvetlov
Copy link
Contributor

left a comment

Anyway, looks good to me

@Carreau
Copy link
Contributor

left a comment

This looks good !

@Carreau

This comment has been minimized.

Copy link
Contributor

commented May 25, 2019

We likely can also have entries in tutorials, and pydoc.py for later.

@1st1 1st1 merged commit 16cefb0 into python:master May 27, 2019

5 checks passed

Azure Pipelines PR #20190523.65 succeeded
Details
bedevere/issue-number Issue number 37028 found
Details
bedevere/news News entry found in Misc/NEWS.d
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@1st1 1st1 deleted the 1st1:arepl branch May 27, 2019

@brandtbucher brandtbucher referenced this pull request Jun 11, 2019

Merged

python -m static_frame #61

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.