Prototype async REPL using IPython, take III

[Carreau]

This is a squash and a rebase of a large number of commits from Min and
I. For simplicity of managing it, history has been reduced to a single
commit, but more historical versions can be found, in particular in
PR #11155, or commit aedb5d6 to be more
exact.

--

I'm expecting some errors in the squash and resolve of conflicts, so will iterate on it.

[Carreau]

I'm thinking of dropping 3.4, as maintaining compat with it will be painful:

There are no specific plans for the next release of Python 3.4. However, the expectation is that Python 3.4.10 will be released in March of 2019, and this will be the final release of Python 3.4.

We'll also be supporting 3.5, 3.6 and 3.7.

@minrk and @takluyver thoughts ?

[minrk]

asyncio support is pretty valuable and long overdue, so if 3.4 makes it too hard, I think the trade-off is worth it.

[minrk]

Neither of these are dependencies, right? We shouldn't require these packages.

[Carreau]

One of them is necessary if we want to have nested embed(). I can make that optional, I probably added them here to have the test passing.

[dalejung]

@Carreau I ran into an issue running python files that have their own loop calls. So if I have a file that calls run_until_complete, it'll break when %run since it nests within the loop_runners own call.

Can we change the run_cell_async coro to first yield interactivity so it can call the pseudo sync when async is not required?

dalejung@8d38b3c

is a working example. await statements in the shell work while running or calling funcs/files that invoke their own loops work as well.

[Carreau]

I have to better understand the implication. It is likely possible, though it will likely work only on terminal IPython.

It might also not stay forever, as in the end I want IPython to always run on an event loop. Though, I guess we could run only on the pseudo-sync, or a non-asyncio one.

Once that hapend we can figure out how to patch %run in order to patch run_until_complete

[njsmith]

The way I would imagine it working in the long run is:

• The async annotations spread out from run_cell to infect the rest of IPython, so that the whole program runs under a single call to the event loop
• By default IPython starts out in 'sync' mode, which means:
  • Using the pseudo-sync runner
  • Raise SyntaxError if someone tries to use a top-level await in a cell
• You can switch from 'sync' mode to 'asyncio' mode, 'trio' mode etc., which switches IPython to using these loops
  • Internally, this loop switcheroo can be accomplished by yielding a special value to the pseudo-sync runner, which then resumes the same coroutine object under the new loop. This is serious black magic but should work...
  • Also it sets a flag internally to start allowing top-level await inside cells
• You can use configuration, pass a command-line flag, etc., to preemptively switch into one of the async modes at startup

Of course there's a ways to go before we use async annotations everywhere like that. But we need to eventually reach a point of keeping the same loop alive over multiple cells, if we want this to be really useful. In any case, I think the basic idea of tracking whether we're in sync or async mode, and turning the magic parser on-or-off as appropriate, is a good one to expose to the user. And that's something that can be done now and would solve @dalejung's problem.

[njsmith]

...of course when writing the above I forgot about prompt-toolkit. Besides it needing some work to integrate with all these different loops, I'm not sure that it'll be happy switching between loops mid-stream...

Well, the architecture I described above ought to work for the kernel, or for jupyter console, at least :-). (With no GUI integration enabled.)

[Carreau]

Thanks ! Much appreciated.

[Carreau]

That would close #9166 as well.

[Carreau]

Todo, catch exception when the runner crash, EG, when using autoawait trio, and then await asyncio.sleep(0).

[EDIT]: Now done.

[Carreau]

Timeit, time, and prun magics will need updates to work with async code.
They still work for sync code though, so I don't think it is a blocker.

[minrk]

deprecations

[Carreau]

Thanks, Fixed.

[minrk]

Adding yield interactivity here makes this a not-entirely-valid coroutine, doesn't it? When I try to call await run_cell_async from ipykernel, I get a bad yield: 'last_expr'. How can we communicate this required info without faking that this isn't actually a coroutine? Does it need to be an actual async generator?

[Carreau]

That's right. You need to do a small dance. I believe I have a patch on my machine for ipykernel to make this work. You need to advance the coro by sending None to it once. We could do it in another way, but at some point we'll always hit the fact that python does not have a real async-exec and that we need to distinguish 2 path of running sync code, or asyc one.

…

On Tue, Sep 4, 2018, 16:43 Min RK ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In IPython/core/interactiveshell.py <#11265 (comment)>: > @@ -2780,9 +2984,14 @@ def error_before_exec(value): self.displayhook.exec_result = result # Execute the user code - interactivity = 'none' if silent else self.ast_node_interactivity - has_raised = self.run_ast_nodes(code_ast.body, cell_name, - interactivity=interactivity, compiler=compiler, result=result) + interactivity = "none" if silent else self.ast_node_interactivity + if _run_async: + interactivity = 'async' + # yield interactivity so let run_cell decide whether to use + # an async loop_runner + yield interactivity Adding yield interactivity here makes this a not-entirely-valid coroutine, doesn't it? When I try to call await run_cell_async from ipykernel, I get a bad yield: 'last_expr'. How can we communicate this required info without faking that this isn't actually a coroutine? Does it need to be an actual async generator? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11265 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUez6-l5irZuy9QnGMdamhrYJsfea77ks5uXpGXgaJpZM4V7qox> .

[Carreau]

Ok, seem like we've reached quasi static state and most of the latest updates were documentation. Plus ipykernel works with that. So I'm going to merge (thanks all involved !), and we can refine if necessary.

[minrk]

Fair enough. I'm just wondering if there's a better way to check if it should be async explicitly first, rather than relying on the fact that coroutines are actually just a special kind of generators, and having to call it in a special way that no other coro can be called.

[minrk]

#11289 makes run_cell_async a regular coroutine by moving the 'should it be async' check to a separate method.

Since in general, running it with a coroutine runner ought to work (I know there are edge cases for now), skipping this check should be safe.