Not Implemented Error Coming from ZMQ/asyncio

[crawftv]

Getting a not implemented error when running execute_notebook.
python 3.8
windows 10
virtualenv
A link to my project I am working on is here. https://github.com/crawftv/crawto.
I tried running it without a separate jupyter instance running but that did not help.

papermill crawto.ipynb crawto1.ipynb
Input Notebook:  crawto.ipynb
Output Notebook: crawto1.ipynb
Executing:   0%|                                                                              | 0/13 [00:00<?, ?cell/s]
Traceback (most recent call last):
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 193, in _run_module_as_main  
    return _run_code(code, main_globals, None,  
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 86, in _run_code  
    exec(code, run_globals)
  File "E:\projects\crawto\.test-env\Scripts\papermill.exe\__main__.py", line 7, in <module>  
  File "e:\projects\crawto\.test-env\lib\site-packages\click\core.py", line 829, in __call__  
    return self.main(*args, **kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\click\core.py", line 782, in main  
    rv = self.invoke(ctx)  
  File "e:\projects\crawto\.test-env\lib\site-packages\click\core.py", line 1066, in invoke  
    return ctx.invoke(self.callback, **ctx.params)  
  File "e:\projects\crawto\.test-env\lib\site-packages\click\core.py", line 610, in invoke  
    return callback(*args, **kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\cli.py", line 222, in papermill  
    execute_notebook(  
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\execute.py", line 95, in  
execute_notebook  
    nb = papermill_engines.execute_notebook_with_engine(  
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\engines.py", line 49, in  execute_notebook_with_engine  
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\engines.py", line 343, in execute_notebook    
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\engines.py", line 402, in execute_managed_notebook  
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()    
  File "e:\projects\crawto\.test-env\lib\site-packages\papermill\clientwrap.py", line 36, in execute  
    with self.setup_kernel(**kwargs):  
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\contextlib.py", line 113, in __enter__  
    return next(self.gen)  
  File "e:\projects\crawto\.test-env\lib\site-packages\nbclient\client.py", line 422, in setup_kernel  
    self.start_new_kernel_client(**kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\nbclient\util.py", line 37, in wrapped  
    result = loop.run_until_complete(coro(self, *args, **kwargs))  
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 616, in run_until_comple  File "e:\projects\crawto\.test-env\lib\site-packages\nbclient\client.py", line 396, in async_start_new_kernel_client  
    await ensure_async(self.kc.start_channels())  
  File "e:\projects\crawto\.test-env\lib\site-packages\jupyter_client\client.py", line 106, in start_channels  
    self.shell_channel.start()  
  File "e:\projects\crawto\.test-env\lib\site-packages\jupyter_client\client.py", line 151, in shell_channel  
    socket = self.connect_shell(identity=self.session.bsession)  
  File "e:\projects\crawto\.test-env\lib\site-packages\jupyter_client\connect.py", line 561, in connect_shell  
    return self._create_connected_socket('shell', identity=identity)  
  File "e:\projects\crawto\.test-env\lib\site-packages\jupyter_client\connect.py", line 545, in _create_connected_socket  
    sock = self.context.socket(socket_type)   
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\sugar\context.py", line 212, in socket  
    s = self._socket_class(self, socket_type, **kwargs)  
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\_future.py", line 144, in __init__
    self._init_io_state()  
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\asyncio\__init__.py", line 53, in _init_io_state  
    self.io_loop.add_reader(self._fd, lambda : self._handle_events(0, 0))  
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 501, in add_reader  
    raise NotImplementedError  
NotImplementedError  
Exception ignored in: <function Socket.__del__ at 0x000001B167FCD820>  
Traceback (most recent call last):  
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\sugar\socket.py", line 67, in __del__  
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\_future.py", line 160, in close  
  File "e:\projects\crawto\.test-env\lib\site-packages\zmq\asyncio\__init__.py", line 60, in  _clear_io_state  
  File "C:\Users\cwcol\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 504, in remove_reader  
NotImplementedError:```

[MSeal]

What version of pyzmq are you running with? This looks like an issue in that library assuming state about asyncio.

[crawftv]

pyzmq==19.0.1
papermill==2.1.1

[MSeal]

Was just going to mention the package is pyzmq even though the module is just zmq but it looks like you worked that out.

[MSeal]

@minrk Do you know if that latest release of pyzmq has async compatibility issues around some of the new socket cleanup code? I didn't see an issue yet.

[minrk]

CPython 3.8 changed the default event loop on Windows to "proactor" which lacks the basic add_reader functionality necessary for being used from pyzmq, tornado, etc. See this issue in tornado.

The recommendation from Python itself was for all applications to set the event loop policy to use the select loop, which was the default prior to 3.8:

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Since there's no good way for libraries to set a policy (it wouldn't be super appropriate to set a global like this at import time), it's on all applications that use these libraries to set this policy themselves prior to creating the event loop.

I don't know if it's feasible for pyzmq to support the proactor APIs, but haven't had time to investigate.

[MSeal]

That does clarify the issue well. I can add a patch to papermill here to cover entrypoint conditions for windows event loop on 3.8 then since this gets launched before most event loops have started.

[MSeal]

I should have time to get a patch in papermill at least in the next day or two. Really annoying that CPython has different async interaction on Windows for 3.8 :/

[MSeal]

This should be resolved in the 2.1.2 release now