semicolon detection attempts to parse anything, not just python #14195
Description
Changes from #14163 seem to cause some parsing code that (if I'm getting it right) is looking for a ; in python code to run on anything, including line magics, at least under some circumstances. This results in spurious exceptions and other weird behavior. Here's a quick MWE that reproduces this for me in ipython 18.16.0. The code below is intended to run in ipython in the command line, and this doesn't seem to affect jupyter lab at all, but it does affect nbconvert (I ran into it with automated testing via nbvconvert).
MWE line magic to get buggy behavior (it seems that the line magic needs to return something):
from IPython.core.magic import register_line_magic
@register_line_magic
def echo(line):
return line
Cases:
This works:
In [2]: %echo test
Out[2]: "test"
This causes the output to be swallowed, I think because of the ;:
In [3]: %echo test;
There are various examples that break this completely, I think because they are trying to parse python and failing (though that doesn't seem to be sufficient). Here is one:
In [7]: %echo test)
---------------------------------------------------------------------------
TokenError Traceback (most recent call last)
Cell In[7], line 1
----> 1 get_ipython().run_line_magic('echo', 'test)')
File ~/miniforge3/envs/lnb2/lib/python3.11/site-packages/IPython/core/displayhook.py:259, in DisplayHook.__call__(self, result)
253 """Printing with history cache management.
254
255 This is invoked every time the interpreter needs to print, and is
256 activated by setting the variable sys.displayhook to it.
257 """
258 self.check_for_underscore()
--> 259 if result is not None and not self.quiet():
260 self.start_displayhook()
261 self.write_output_prompt()
File ~/miniforge3/envs/lnb2/lib/python3.11/site-packages/IPython/core/displayhook.py:87, in DisplayHook.quiet(self)
85 """Should we silence the display hook because of ';'?"""
86 if self.exec_result is not None:
---> 87 return self.semicolon_at_end_of_expression(self.exec_result.info.raw_cell)
88 return False
File ~/miniforge3/envs/lnb2/lib/python3.11/site-packages/IPython/core/displayhook.py:95, in DisplayHook.semicolon_at_end_of_expression(expression)
92 """Parse Python expression and detects whether last token is ';'"""
94 sio = _io.StringIO(expression)
---> 95 tokens = list(tokenize.generate_tokens(sio.readline))
97 for token in reversed(tokens):
98 if token[0] in (tokenize.ENDMARKER, tokenize.NL, tokenize.NEWLINE, tokenize.COMMENT):
File ~/miniforge3/envs/lnb2/lib/python3.11/tokenize.py:525, in _tokenize(readline, encoding)
523 else: # continued statement
524 if not line:
--> 525 raise TokenError("EOF in multi-line statement", (lnum, 0))
526 continued = 0
528 while pos < max:
TokenError: ('EOF in multi-line statement', (2, 0))