nest_asyncio broke something in nbclient

[qci-amos]

my unit tests are now giving the exception:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/testbook/testbook.py:52: in __enter__
    self._prepare()
/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/testbook/testbook.py:46: in _prepare
    self.client.execute()
/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/testbook/client.py:147: in execute
    super().execute_cell(cell, index)
/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/nbclient/util.py:74: in wrapped
    return just_run(coro(*args, **kwargs))
/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/nbclient/util.py:53: in just_run
    return loop.run_until_complete(coro)
/home/amos/miniconda3/envs/aq/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
    return future.result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testbook.client.TestbookNotebookClient object at 0x7fc0bcd87670>
cell = {'cell_type': 'code', 'execution_count': None, 'id': '6765142a', 'metadata': {'execution': {}}, 'outputs': [], '<snip>
cell_index = 6, execution_count = None, store_history = True

    async def async_execute_cell(
            self,
            cell: NotebookNode,
            cell_index: int,
            execution_count: t.Optional[int] = None,
            store_history: bool = True) -> NotebookNode:
        """
        Executes a single code cell.

        To execute all cells see :meth:`execute`.

        Parameters
        ----------
        cell : nbformat.NotebookNode
            The cell which is currently being processed.
        cell_index : int
            The position of the cell within the notebook object.
        execution_count : int
            The execution count to be assigned to the cell (default: Use kernel response)
        store_history : bool
            Determines if history should be stored in the kernel (default: False).
            Specific to ipython kernels, which can store command histories.

        Returns
        -------
        output : dict
            The execution output payload (or None for no output).

        Raises
        ------
        CellExecutionError
            If execution failed and should raise an exception, this will be raised
            with defaults about the failure.

        Returns
        -------
        cell : NotebookNode
            The cell which was just processed.
        """
        assert self.kc is not None
        if cell.cell_type != 'code' or not cell.source.strip():
            self.log.debug("Skipping non-executing cell %s", cell_index)
            return cell

        if self.record_timing and 'execution' not in cell['metadata']:
            cell['metadata']['execution'] = {}

        self.log.debug("Executing cell:\n%s", cell.source)

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or "raises-exception" in cell.metadata.get("tags", []))

        parent_msg_id = await ensure_async(
            self.kc.execute(
                cell.source,
                store_history=store_history,
                stop_on_error=not cell_allows_errors
            )
        )
        # We launched a code cell to execute
        self.code_cells_executed += 1
        exec_timeout = self._get_timeout(cell)

        cell.outputs = []
        self.clear_before_next_output = False

        task_poll_kernel_alive = asyncio.ensure_future(
            self._async_poll_kernel_alive()
        )
        task_poll_output_msg = asyncio.ensure_future(
            self._async_poll_output_msg(parent_msg_id, cell, cell_index)
        )
        self.task_poll_for_reply = asyncio.ensure_future(
            self._async_poll_for_reply(
                parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
            )
        )
        try:
            exec_reply = await self.task_poll_for_reply
        except asyncio.CancelledError:
            # can only be cancelled by task_poll_kernel_alive when the kernel is dead
            task_poll_output_msg.cancel()
>           raise DeadKernelError("Kernel died")
E           nbclient.exceptions.DeadKernelError: Kernel died

/home/amos/miniconda3/envs/aq/lib/python3.8/site-packages/nbclient/client.py:845: DeadKernelError

which goes away if I bring nest_asyncio back to 1.5.1

[erdewit]

   except asyncio.CancelledError:
       ...
      raise DeadKernelError("Kernel died")

The cause seems to be a cancelation. Can this be related to the fix for your #58 issue?

[qci-amos]

I don't test the combination of jupyter + cancellation -- these tests don't do cancel

• I've confirmed the error goes away if I downgrade to 1.5.1
• the context for the trace above was a pytest using testbook, but I've confirmed that an interactive notebook fails too, it looks like right after it successfully executes the last cell

these were the only logs from jupyter:

[I 14:00:12.410 NotebookApp] Replaying 3 buffered messages
[I 14:01:28.417 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel 59511ee4-8c2c-419f-8ba7-25f2b4d14a43 restarted
[I 14:02:04.646 NotebookApp] Saving file at /jupyter_tests.ipynb

[qci-amos]

this doesn't show the same issue, but it might be related:

[erdewit]

I can't reproduce the error with the task being None, but there's a new v1.5.4 version out that might fix it.

[qci-amos]

yes, I can confirm the latest code changes address my problem. thank you!

[erdewit]

OK that's great to hear.