New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't invoke asyncio event_loop after tornado 5.0 update #3397

Closed
betafcc opened this Issue Mar 5, 2018 · 36 comments

Comments

Projects
None yet
@betafcc
Copy link

betafcc commented Mar 5, 2018

On fresh python3.6 venv, after pip install jupyter && jupyter notebook and starting a new python3.6 notebook:

import asyncio

async def foo():
    return 42

asyncio.get_event_loop().run_until_complete(foo())

throws:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-3ad9bf216544> in <module>()
----> 1 asyncio.get_event_loop().run_until_complete(foo())

/usr/local/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
    452         future.add_done_callback(_run_until_complete_cb)
    453         try:
--> 454             self.run_forever()
    455         except:
    456             if new_task and future.done() and not future.cancelled():

/usr/local/lib/python3.6/asyncio/base_events.py in run_forever(self)
    406         self._check_closed()
    407         if self.is_running():
--> 408             raise RuntimeError('This event loop is already running')
    409         if events._get_running_loop() is not None:
    410             raise RuntimeError(

RuntimeError: This event loop is already running

If I specify tornado==4.5.3 before pip install jupyter, it works fine

@betafcc betafcc changed the title Can't invoke asyncio event_loop after tornado update Can't invoke asyncio event_loop after tornado 5.0 update Mar 5, 2018

@lschoe

This comment has been minimized.

Copy link

lschoe commented Mar 6, 2018

Same for me (fresh Python 3.6.4 plus Jupyter, on Windows 7 32-bit).

@betafcc

This comment has been minimized.

Copy link
Author

betafcc commented Mar 6, 2018

Probably related to the fact that

"On Python 3, IOLoop is always a wrapper around the asyncio event loop."

as listed in "Backwards-compatibility notes" of tornado 5.0: http://www.tornadoweb.org/en/stable/releases/v5.0.0.html#backwards-compatibility-notes

@takluyver

This comment has been minimized.

Copy link
Member

takluyver commented Mar 6, 2018

I'm not sure at the moment what we can do about this; as the error message says, the event loop is already running, because tornado now runs on top of asyncio. What you want is essentially await, but you can't use that outside a function.

@Carreau I think you were working on something related to this. Any ideas?

@lschoe

This comment has been minimized.

Copy link

lschoe commented Mar 6, 2018

Guess Tornado needs to use its own event loop coupled to a different thread than the one(s) used for the Jupyter notebook cells. Python 3.6 documentation mentions in Section 18.5.2.4 that "The default policy defines context as the current thread, and manages an event loop per thread that interacts with asyncio." Maybe people have already experimented with this?

@takluyver

This comment has been minimized.

Copy link
Member

takluyver commented Mar 6, 2018

I'd be inclined to figure out a way to run the user's coroutines on the existing event loop rather than starting a new thread. Threads cause all sorts of problems.

@Carreau

This comment has been minimized.

Copy link
Contributor

Carreau commented Mar 6, 2018

Yep, ipython/ipython#10390 should help, but I din't had much chance to work on it. One of the TODO items is to make it work with ipykernel and run things on the current eventloop.

The framework is there it probably does not need much changes to work.

@mingwandroid

This comment has been minimized.

Copy link

mingwandroid commented Mar 27, 2018

Hey all, just wondering if there is there any update on this issue?

@parmentelat

This comment has been minimized.

Copy link
Contributor

parmentelat commented Mar 28, 2018

I'd like to second this request for an update

As far as I am concerned, this is a major hindrance, as anything remotely useful tends to have some dosage of asyncio these days

@parmentelat

This comment has been minimized.

Copy link
Contributor

parmentelat commented Mar 28, 2018

At the risk of stating the obvious, I'd just like to outline the following discrepency between what I get in a terminal - be it python or ipython - and in a notebook

I am not sure that I quite understand the other discussion there ipython/ipython#10390 but regardless, it is my feeling that something is wrong at this very early point already.

screen shot 2018-03-28 at 09 42 28

image

@takluyver

This comment has been minimized.

Copy link
Member

takluyver commented Mar 28, 2018

Yep, that's expected. The kernel itself runs on an event loop, and as of Tornado 5.0, it's using the asyncio event loop. So the asyncio event loop is always running in the kernel. As far as I know, we haven't figured out a way to deal with this yet.

@parmentelat

This comment has been minimized.

Copy link
Contributor

parmentelat commented Mar 28, 2018

but wait, as far as I am concerned this means I can't run anything tainted with asyncio in a notebook

which will asymptotically amount to saying, I can't run anything in a notebook ;)

it that right, or am I missing something obvious ?

@takluyver

This comment has been minimized.

Copy link
Member

takluyver commented Mar 28, 2018

For the moment, that's about right, though I don't think it's ever going to be the case that everything is async. We need to work out a way around it, but so far we haven't got there.

@minrk

This comment has been minimized.

Copy link
Member

minrk commented Mar 28, 2018

Right, you can't instantiate and run an asyncio eventloop in a thread with an asyncio loop already running. You can run asyncio in a thread, as seen in #11030, which is tedious and we would like it to be unnecessary. That's going to require that we finish #10390 or similar.

You can pin tornado to 4.x while we figure this out.

@btoueg

This comment has been minimized.

Copy link

btoueg commented Mar 29, 2018

I'm facing the same issue while I'm trying to run some code that uses internally:

with closing(asyncio.new_event_loop()) as loop:
    asyncio.set_event_loop(loop)
    return loop.run_until_complete(get_converted_form(msg))

Jupyter gives: Cannot run the event loop while another loop is running

@parmentelat

This comment has been minimized.

Copy link
Contributor

parmentelat commented Mar 29, 2018

to make min's suggestion more concrete, I could work around this issue by just issuing

pip3 install tornado==4.5.3

edit: I expect the jupyter server needs to be restarted as well

@takluyver

This comment has been minimized.

Copy link
Member

takluyver commented Mar 29, 2018

You'll need to restart the kernel - you don't have to restart the whole notebook server.

@andyljones

This comment has been minimized.

Copy link

andyljones commented Apr 7, 2018

I also discovered this because all my asyncio code stopped working after an otherwise innocuous update. For anyone else who comes across this while not quite understanding how event loops work: creating a new event loop won't help, as only one can run at any one time.

Some Google bait:

Spyder IPython Jupyter tornado
RuntimeError: Cannot run the event loop while another loop is running
RuntimeError: This event loop is already running
<_UnixSelectorEventLoop running=True closed=False debug=False>
Can't kill event loop

@jar349 jar349 referenced this issue Apr 12, 2018

Closed

auth error #795

@simsicon

This comment has been minimized.

Copy link

simsicon commented Apr 15, 2018

I think a possible solution/workaround is to to use the ioloop of tornado, instead of running a new ioloop, but I did not get it right, any ideas?

@Skorpeo

This comment has been minimized.

Copy link

Skorpeo commented Apr 15, 2018

I have the same issue...the only way I was able to make it work was:

import asyncio
loop = asyncio.get_event_loop()
loop.create_task(some_async_func())

heart = broken

@AIGyan

This comment has been minimized.

Copy link

AIGyan commented Apr 23, 2018

@parmentelat
pip3 install tornado==4.5.3 solved the issue

vdboor added a commit to vdboor/python-zeep that referenced this issue Apr 26, 2018

vdboor added a commit to vdboor/python-zeep that referenced this issue Apr 26, 2018

vdboor added a commit to vdboor/python-zeep that referenced this issue Apr 26, 2018

vdboor added a commit to vdboor/python-zeep that referenced this issue Apr 26, 2018

@minrk minrk referenced this issue May 24, 2018

Merged

async support #323

@getzze

This comment has been minimized.

Copy link

getzze commented Aug 9, 2018

It is working correctly with python3.7, Tornado 5.1 and ipykernel 4.8.2 on ArchLinux by manually updating python-ipykernel to the last version (not working with python-ipykernel-4.6.1)

@erdewit

This comment has been minimized.

Copy link

erdewit commented Aug 14, 2018

@getzze

It is working correctly with python3.7, Tornado 5.1 and ipykernel 4.8.2 on ArchLinux by manually updating python-ipykernel to the last version (not working with python-ipykernel-4.6.1)

Running the code snippet in the first post of this thread with Python 3.7.0, Tornado 5.1 and ipykernel 4.8.2 still gives me the same "RuntimeError: This event loop is already running".

Using the new asyncio.run method instead of run_until_complete gives a slightly different "RuntimeError: asyncio.run() cannot be called from a running event loop".

The real problem is imho not with jupyter or tornado, but with this decision for asyncio.

@Carreau

This comment has been minimized.

Copy link
Contributor

Carreau commented Aug 14, 2018

We are aware of the issue and working on making it easier to run async code within a notebook, without having to manipulate the event loop yourself.

Getting it to work was already challenging, having it function on multiple python version is far from being easy, and we lack people willing to test our in-progress pull request to give feedback. In particular on the IPython side:

And ipykernel side:

There a a lot of extremely subtle behavior,
autoasync

Any help to do other tasks unrelated to this bug might give us some band with to focus on this, but it's relatively complex code that requires multiple hours of focused time, which is becoming rare.

@basnijholt

This comment has been minimized.

Copy link

basnijholt commented Sep 3, 2018

The new Jupyterhub has:

jupyterhub 0.9.2 has requirement tornado>=5.0, but you'll have tornado 4.5.3 which is incompatible.

after which I am unable to install the older version of tornado in the main environment of the Jupyterhub.

Does this mean that until this issue is solved, the root/main environment of any Jupyterhub>=0.9.2 (at least) is broken?

@gusutabopb

This comment has been minimized.

Copy link

gusutabopb commented Sep 3, 2018

@Carreau Those integrations look really nice, but will they be strictly optional? In other words, will there be a way to have the asyncio loop NOT running at all on Jupyter Notebook going forward?

I use Jupyter as a scratchpad for development and as a developer of asyncio-based apps, not being able to run loop.run_forever or loop.run_until_complete (or the new asyncio.run in 3.7) from Jupyter pretty much makes it useless as a scratchpad for me. I am sticking to Python 3.6.6 with tornado 4.5.3 for now, but that clearly isn't sustainable... (conda won't let me install Python 3.7 with tornado 4.5.3)

@Carreau

This comment has been minimized.

Copy link
Contributor

Carreau commented Sep 3, 2018

@jhrmnn

This comment has been minimized.

Copy link

jhrmnn commented Sep 3, 2018

Most asyncio packages have an entry point that deals with the loop. Asyncio itself has asyncio.run() which won't work in the Ipython kernel at the moment. Aiohttp has web.run_app(). Other packages have other functions. These functions set up the loop, run the coroutines, and do application-specific nontrivial teardowns of the loop. Currently, one would have to go one step lower, and reimplement these methods in the Notebook. This is not feasible.

@gusutabopb

This comment has been minimized.

Copy link

gusutabopb commented Sep 4, 2018

@azag0 Exactly. Having the asyncio loop on the main thread always running basically breaks most asyncio-based apps/libraries out there.
Any new comer to the asyncio world that tries to run the code of some asyncio tutorial on Jupyter will be incredibly frustrated as most sample code they find on the web (which usually uses loop.run_until_complete or loop.run_forever) just simply won't run.

@Carreau I am not familiar with the internals of ipykernel, but is it possible to have the ipykernel/tornado asyncio loop running in a background thread instead of the main thread?

@gusutabopb

This comment has been minimized.

Copy link

gusutabopb commented Sep 4, 2018

Quick update on conda:

While conda won't allow downgrading to tornado==4.5.3 on Python 3.7:

$ conda create -y -n py37_3 python=3.7 ipykernel tornado=4.5.3
# UnsatisfiableError: The following specifications were found to be in conflict [...]

I have confirmed overriding tornado through pip does work:

$ conda create -y -n py37 python=3.7 ipykernel  # installs tornado 5.1
$ conda activate py37
$ pip install "tornado==4.5.3"  # Successfully installed tornado-4.5.3

I am not sure if this will cause some minor bugs elsewhere or if the conda dependency check is just being overly strict though.

@Carreau

This comment has been minimized.

Copy link
Contributor

Carreau commented Sep 5, 2018

While conda won't allow downgrading to tornado==4.5.3 on Python 3.7:

That probably because some of the dependencies have not (yet) been packaged on conda. They might be compatible with Python 3.7, but the conda package should be rebuilt.

I see:

UnsatisfiableError: The following specifications were found to be in conflict:
  - python=3.7
  - tornado=4.5.3 -> python=3.5 -> readline=6.2
  - tornado=4.5.3 -> python=3.5 -> sqlite=3.9
  - tornado=4.5.3 -> python=3.5 -> tk=8.5

Packaging for old Python takes time, and I guess that's were companies that rely on that could help, but it's hard to ask volunteers to do that. The other possibility is to get a contract with Anaconda and say you are looking for long term support of X,Y,Z on newer Python.

I am not familiar with the internals of ipykernel, but is it possible to have the ipykernel/tornado asyncio loop running in a background thread instead of the main thread?

As many things in open-source the answer is maybe, but we also have the issues that some stuff don't like not being on the main thread. It's a matter of finding the resource to implement/review/maintain.

I'm tempted to also say that asyncio is cooperative scheduling so these applications should give option to run on already existing loops. I understand the use case, it is just hard to figure out how to do the right thing automatically.

@jhrmnn

This comment has been minimized.

Copy link

jhrmnn commented Sep 5, 2018

I'm tempted to also say that asyncio is cooperative scheduling so these applications should give option to run on already existing loops. I understand the use case, it is just hard to figure out how to do the right thing automatically.

I agree on that point. For instance, with Aiohttp, rather then the blocking web.run_app(app), one can do

runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner)  # web server launched as a background task
await site.start()
# user code
await runner.cleanup()

Ideally, all asyncio Python packages should allow such use. Unfortunately, one cannot turn a package that doesn't provide such functionality into a package that does easily. Even better, packages would provide an async context manager:

async with web.AppRunner(app) as runner:
    async with web.TCPSite(runner) as site:
        # user code

If this was common, the current situation with the Ipython kernel wouldn't really be such an issue.

@mingwandroid

This comment has been minimized.

Copy link

mingwandroid commented Sep 5, 2018

Packaging for old Python takes time, and I guess that's were companies that rely on that could help, but it's hard to ask volunteers to do that.

We can make some old builds if they help the community and there's a good argument for that in this case. I had hoped this issue would be fixed quickly but I don't know anything about the codebase so that hope wasn't based on much! @csoja should we build these packages?

@Carreau

This comment has been minimized.

Copy link
Contributor

Carreau commented Sep 5, 2018

If this was common, the current situation with the IPython kernel wouldn't really be such an issue.

Well it might be a chicken-or-egg problem, they don't provide the functionality because users don't ask. Maybe reaching to them and saying that you need an async entry point could help.

@erdewit

This comment has been minimized.

Copy link

erdewit commented Sep 7, 2018

For those interested, I've just a created package called nest_asyncio that solves the problem by patching asyncio to allow nested event loops. To use it in a notebook is a matter of putting

import nest_asyncio
nest_asyncio.apply()

somewhere near the top.

@minrk

This comment has been minimized.

Copy link
Member

minrk commented Sep 7, 2018

We should have prereleases of IPython 7 and ipykernel 5 next week that will enable top-level async/await in IPython or a notebook.

Because ipykernel itself relies on asyncio, if we want to achieve the ability for users to call get_event_loop().run_until_complete(), we have just a few options:

  1. instruct users to use something like nest_asyncio (neat!)
  2. use nest_asyncio by default in ipykernel (probably simplest, but I can't speak to how robust it is yet, since it seems to have only existed for a few hours)
  3. get off the mainloop, and run all of ipykernel in a background thread. We already do this for IOPub, and we could do it for the rest.

Personally, I view the fact that asyncio and tornado are running to be a feature, as user code can launch long-running coroutines on the main eventloop and they will keep running in the background. This would be challenging if the loop were in a thread.

So my inclination is for now, recommend nest_asyncio at the user-level and finish shipping ipykernel 5 / ipython 7 with top-level await / mainloop. We could even add a special exception handler for this RuntimeError to point folks at nest_asyncio or autoawait:

def recommend_nest_asyncio(shell, etype, value, tb, tb_offset=0):
    shell.showtraceback((etype, value, tb), tb_offset=tb_offset)
    if "already running" in str(value):
        print("""
    You might want to check out `nest_asyncio`, which enables asyncio.run_until_complete() even though asyncio is already running.
    Or with IPython 7, you can `await future` at the top-level without calling `run_until_complete`.
""")
get_ipython().set_custom_exc((RuntimeError,), handle_runtime)

And then revisit the possibility of putting the main eventloop into a background thread at a future date.

@minrk minrk added this to the Not Notebook milestone Sep 13, 2018

thiagoalmeidasa added a commit to thiagoalmeidasa/serenata-toolbox that referenced this issue Oct 31, 2018

eugene-s added a commit to eugene-s/docker-celery-flower that referenced this issue Dec 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment