Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling of repeated NEWLINE tokens #12057

merged 1 commit into from Dec 27, 2019


Copy link

miedzinski commented Dec 22, 2019

Because of cpython's issue 37621 it is possible that tokenizer will emit repeated NEWLINE tokens despite that language specification states it should never happen.

Currently IPython doesn't handle such situation well, e.g.

In [1]: def f(): 
   ...:     x = 0 
   ...:     \ 
Unhandled exception in event loop:
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/eventloop/", line 154, in _run_task
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/eventloop/", line 115, in new_func
    return func(*a, **kw)
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/application/", line 562, in read_from_input
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/key_binding/", line 273, in process_keys
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/key_binding/", line 180, in _process
    self._call_handler(matches[-1], key_sequence=buffer[:])
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/key_binding/", line 323, in _call_handler
  File "/Users/dominik/src/ipython/venv/lib/python3.7/site-packages/prompt_toolkit/key_binding/", line 78, in call
    return self.handler(event)
  File "/Users/dominik/src/ipython/IPython/terminal/", line 128, in newline_or_execute
    status, indent = shell.check_complete(check_text)
  File "/Users/dominik/src/ipython/IPython/core/", line 3361, in check_complete
    status, nspaces = self.input_transformer_manager.check_complete(code)
  File "/Users/dominik/src/ipython/IPython/core/", line 680, in check_complete
    if tokens_by_line[-1][-1].string == ':':

Exception list index out of range
Press ENTER to continue...

This PR attempts to fix this issue.

@Carreau Carreau requested a review from takluyver Dec 27, 2019
@Carreau Carreau added this to the 7.11 milestone Dec 27, 2019
@Carreau Carreau force-pushed the miedzinski:linecont branch from b758b5b to 0c5d00e Dec 27, 2019
@Carreau Carreau merged commit 1606449 into ipython:master Dec 27, 2019
2 checks passed
2 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
continuous-integration/travis-ci/pr The Travis CI build passed

This comment has been minimized.

Copy link

Carreau commented Dec 27, 2019

Thanks, I had to rebase to get the test-suite passing.

This should be released soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can’t perform that action at this time.