Ellipsis as dictionary literal key causes Syntax Error

[betafcc]

Probably too specific to receive attention but maybe it highlights some internal design flaw or something:

Both in IPython console and Jupyter notebook cells (with ipython kernel), the following happens:

• opening brace in same line as ellipsis works fine

In [1]: { ...: 42 }
Out[1]: {Ellipsis: 42}

• ellipsis in beginning of line, without space before colon doesn't get recognized (note the set({42}) returned)

In [2]: {
   ...: ...: 42
   ...: }
Out[2]: {42}

• ellipsis in beginning of line, with space before colon causes SyntaxError

In [3]: {
   ...: ... : 42
   ...: }
  File "<ipython-input-3-eb9728c6c108>", line 2
    : 42
    ^
SyntaxError: invalid syntax

• ellipsis after spaces, with space before colon works fine

In [4]: {
   ...:     ... : 42
   ...: }
Out[4]: {Ellipsis: 42}

• ellipsis after spaces, without space before colon doesn't get recognized

In [5]: {
   ...:     ...: 42
   ...: }
Out[5]: {42}

• with any items before, it always work fine

In [6]: {
   ...:     'foo': 'bar', ...: 42
   ...: }
Out[6]: {'foo': 'bar', Ellipsis: 42}

• but items aftwards doesn't make a difference on previous cases

In [7]: {
   ...:     ...: 42, 'foo': 'bar'
   ...: }
  File "<ipython-input-7-449ccd03925d>", line 2
    42, 'foo': 'bar'
             ^
SyntaxError: invalid syntax

And all these cases works fine in a source file ot in normal python repl (tested 3.5, 3.6, 3.7 and IPython 6.5.0)

I realize it's a very specific bug with an easy workaround, and it's probably similar to #2605 and #4059 , but I was working in jupyter using a match(obj, cases) function (eg: match(foo(), {'bar' : 42}), using ... for the default case and this bug tripped me so hard before I started to understand, it was worthy of this issue

[Carreau]

I believe I see where this is coming from.
We try to do some magic to let user past code that already have an IPython prompt in the front.

@takluyver did work on the tokinsation, so may have some more insight, and does have plan to refactor.

Thanks for the exaustive use case, that would be of great help when we work on this.

[takluyver]

I suspect it's hard to avoid that totally given what we want to do. ... is recognised as the continuation prompt from the plain Python interactive shell; we strip that off so you can easily copy/paste examples that have the prompts included. We probably started doing that before ... became valid Python syntax.

We could decide to stop doing that; it would in some senses be more 'correct' not to, because it can mangle valid Python code, as you found. On the other hand, that's still what the prompts in the Python shell look like, I still see examples including the prompts, and there are an unknown number of people using this feature to copy-paste examples without having to remove all the prompts manually. So maybe we just have to write this off as a corner case.

[betafcc]

I agree it's an unimportant corner case, most people don't even know ... is valid syntax at all

Also, didn't know I could paste code with prompt 😮 definitely more valuable

[betafcc]

Stumbled on this again today, even in strings that looks like console input are stripped:

Because of my previous problem, I learned why, ipython applies the PromptStripper which receives each cell splitted by lines right? If someone else stumble on this, here's the workaround I'm using:

That is, just remove it by running get_ipython().input_transformers_cleanup.pop(1), 'input_transformers_cleanup' is just an array with the functions that will be run in each line of user input, I'm not sure if the PromptStripper is always in index '1' tho, a bit more safe would be using:

from IPython.core.inputtransformer2 import PromptStripper

fs = get_ipython().input_transformers_cleanup
fs.remove(next(f for f in fs if isinstance(f, PromptStripper)))

The elipsis case of my previous problem will work also

[betafcc]

btw @takluyver now I think It's possible to keep the 'raw console paste' functionality and fix these cases, at least if there is a preprocess step done 'by cell' instead of 'by line' (which I guess there is right? How does autoawait work? Wraps each line in functions?).

Just made a cell magic as POC, that simply uses ast.parse to check for syntax error before checking for leading '>>>' and '...' to strip, probably there are lighter ways to check it, like what is done in doctest.DocTestFinder but I am still a total newbie in ipython and a bit in python in general so I can't reach a PR-worthy solution

edit

Just noticed it actually works by cell already, so the POC is:

import ast
from IPython.core.inputtransformer2 import PromptStripper

prev = PromptStripper.__call__

def wrapper(self, lines):
    try:
        ast.parse(''.join(lines))
        return lines
    except SyntaxError:
        return prev(self, lines)

PromptStripper.__call__ = wrapper

seems to work: