Nothing works in embedded mode after moving from 8.10 to 8.11

[akhileshraju]

Issue

I am embedding IPython in my command line application. v8.5 to v8.10 work fine but after upgrading to v8.11 nothing works.
The command line starts up, but every command results in Exception 'NoneType' object has no attribute 'check_complete'

Simple act of starting up the application and pressing 'Enter' also throws the error.

Exception

***************************
 <Banner>
***************************


Unhandled exception in event loop:
  File "/home/user1/.pyenv/versions/3.11.1/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/input/vt100.py", line 173, in callback_wrapper
    callback()
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 707, in read_from_input
    self.key_processor.process_keys()
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py", line 270, in process_keys
    self._process_coroutine.send(key_press)
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py", line 185, in _process
    self._call_handler(matches[-1], key_sequence=buffer[:])
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py", line 320, in _call_handler
    handler.call(event)
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_bindings.py", line 124, in call
    result = self.handler(event)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/IPython/terminal/shortcuts/__init__.py", line 374, in handle_return_or_newline_or_execute
    return newline_or_execute_outer(shell)(event)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/dev/cli/.venv/lib/python3.11/site-packages/IPython/terminal/shortcuts/__init__.py", line 397, in newline_or_execute
    status, indent = shell.check_complete(check_text)
                     ^^^^^^^^^^^^^^^^^^^^

Exception 'NoneType' object has no attribute 'check_complete'
Press ENTER to continue...

Application script

import os
import tempfile
from pathlib import Path
import argparse

from traitlets.config.loader import Config
from IPython.terminal.embed import InteractiveShellEmbed


parser = argparse.ArgumentParser()
parser.add_argument("-s", "--source", help="source the python script file", type=str)
args = parser.parse_args()

cfg = Config()

ipshell = InteractiveShellEmbed(
    config=cfg,
    banner1="\n***************************\n <Banner> \n***************************\n",
    exit_msg="\nExiting...",
)

banner = (
    "#####################################################################\n"
    "# Dummy banner \n"
    "#####################################################################\n"
)

if os.access(Path.cwd(), os.W_OK):
    log_path = Path.cwd()
else:
    log_path = Path(tempfile.gettempdir())

ipshell.logger.logstart(
    loghead=banner,
    logfname=str(log_path.joinpath("log.py")),
    logmode="over",
    log_output=True,
)

ipshell()

[krassowski]

Thank you for the report, apologies for the regression. I can reproduce the problem with your script. This likely has to do with get_ipython not returning the shell correctly.

When I tried to create a smaller minimal reproducible example with:

from IPython.terminal.embed import embed; embed()

It appears to work just fine, so it seems to be specific to the direct use of InteractiveShellEmbed rather than the embed mode provided by embed() function.

This works:

from traitlets.config.loader import Config
from IPython.terminal.embed import embed

cfg = Config()

embed(
    config=cfg,
    banner1="\n***************************\n <Banner> \n***************************\n",
    exit_msg="\nExiting...",
)

This does not:

from traitlets.config.loader import Config
from IPython.terminal.embed import InteractiveShellEmbed

cfg = Config()

InteractiveShellEmbed(
    config=cfg,
    banner1="\n***************************\n <Banner> \n***************************\n",
    exit_msg="\nExiting...",
)()

It is not obvious to me what causes this difference.

[samuelsadok]

Minimal reproducable example:

import IPython; console = IPython.terminal.embed.InteractiveShellEmbed(); console()

The error can be seen in a fresh virtual environment:

python -m venv venv
source venv/bin/activate
pip install ipython
python -c "import IPython; console = IPython.terminal.embed.InteractiveShellEmbed(); console()"
# [press enter]

As mentioned by the OP, a downgrade (e.g. pip install "ipython<=8.10") fixes the issue.

In our application we do a bunch of additional setup on the InteractiveShellEmbed object before launching the console, so I think we can't use embed() directly.

Update

After digging a bit deeper, the issue seems to be introduced by 875ff23 by adding a dependency to IPython.core.getipython.get_ipython() via handle_return_or_newline_or_execute(). However when instantiating InteractiveShellEmbed manually, it is not automatically set as the current shell instance.

To fix this, instead of calling InteractiveShellEmbed(...), one can call InteractiveShellEmbed.instance(...) (which is what embed() does).

So this works fine for me, but in terms of suggested fixes to IPython maybe it would be cleaner/safer to have the instance be set automatically when launching (rather than instantiating) the console?

Update 2

Seems the suggested workaround doesn't actually work on Windows (or generally when starting the script with ipython instead of python). So we need to guard the workaround behind an if:

if IPython.core.getipython.get_ipython() is None:
    console = IPython.terminal.embed.InteractiveShellEmbed.instance()
else:
    console = IPython.terminal.embed.InteractiveShellEmbed()

[akhileshraju]

@samuelsadok thanks for the detailed analysis and workaround.

@krassowski is there a plan to fix this in a newer version of IPython? I imagine this is fix-worthy since it's a regression.

[krassowski]

Yes, I think it should be fixed because current examples are broken. After some thinking, I would say instance() should be set in __call__ method of InteractiveShellEmbed as this is intended to activate the shell. More generally I feel like it should override the current instance if there is one, and on exit it should restore it (so basically we should maintain a stack of instances to allow nested embedding, while enforcing that the currently active instance is bound to the singleton tracking the current instance).

ipython/IPython/terminal/embed.py

Line 186 in d016fdb

def __call__(

Any thoughts on such a solution?

[shayneoneill]

note, I get this same bug just by using the ipython command line.

On all my machines.

[akhileshraju]

Hi @krassowski any updates on the fix for this issue? I didn't reply to your earlier question since I am not really an expert on the internals of IPython and didn't want to give any incorrect answers.

[akhileshraju]

Since there doesn't seem to be any movement on this, I am assuming this won't be fixed.
Closing this since the workaround SeismicSource/sourcespec@8bde65c works for me