Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes 11334/11354 Multi line code blocks #11358

Merged
merged 17 commits into from
Oct 8, 2018
Merged

Fixes 11334/11354 Multi line code blocks #11358

merged 17 commits into from
Oct 8, 2018

Conversation

tonyfast
Copy link
Contributor

@tonyfast tonyfast commented Oct 4, 2018

This pull request merges the recent input transformer changes from master. It builds off of the work by @bartskowron in #11354 to resolve #11334 . The changes pass tests locally, including the new tests provided by @Carreau; let's see what CI says.

@takluyver I commented out an assertion. I am not sure what the statement, is this change ok?

bxsx and others added 8 commits October 2, 2018 00:34
@bxsx
Fix ipython#11334: Cannot make multi-line code blocks in ipython
a6c064a 
When codeop.compile_command() returns None it actually says "at least
some part of the code was compiled successfully" which is not really important
for checking if it's complete or not.
Once we haven't got any errors during compilation process, we just want
to check if there will be another nested block of code or not by checking
a colon.
@bxsx
Fix regression...
0a42b86
@bxsx
Add a new regression test and fix for it
fde3f77
@bxsx
Remove unused assignment
d0cad1f
@Carreau
add tests
043b677
@bxsx
always add a trailing newline
498f5f7
@tonyfast
Merge branch 'master' into pr/11354
0e3f99c
@tonyfast
Add some logic to pass all of the check_complete tests
186524b 
An assert statement needed to be commented out.  I'm not sure what the statement was stating though.
@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 4, 2018

Anyone have any ideas why this may be passing on everything, but Python 3.7dev?

@Carreau
Copy link
Member

Carreau commented Oct 4, 2018

Anyone have any ideas why this may be passing on everything, but Python 3.7dev?

Maybe a change of API in generate_token or similar on Pyhton 3.7 dev.

@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 4, 2018

I'll scope it out. Thanks.

@Carreau
Copy link
Member

Carreau commented Oct 4, 2018

that's https://bugs.python.org/issue33899

@Carreau
Copy link
Member

Carreau commented Oct 4, 2018 

## 3.6
In [8]: list(generate_tokens(iter('a = 1').__next__))
Out[8]:
[TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

AFAICT we need need to add a \n (NEWLINE) before ENDMARKER

@Carreau Carreau requested a review from takluyver October 4, 2018 03:26
@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 4, 2018

Thanks for these references @Carreau . I added some conditions for the newline tokens. Let's see if this helps.

@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 4, 2018 •
edited
Loading

The CPython changes to tokenize effect 3.6 & 3.7. The IPython tests are not checking on Python3.6dev.
Should there be tests for 3.6dev because of this use case?

I'm currently working on this pr on py37rc1.

@Carreau
Copy link
Member

Carreau commented Oct 4, 2018 via email

@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 4, 2018 •
edited
Loading

If we consider a simple statement like a=999.

In Python 3.6 with the old tokenize:

Version info sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)
[TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a=999'),
 TokenInfo(type=53 (OP), string='=', start=(1, 1), end=(1, 2), line='a=999'),
 TokenInfo(type=2 (NUMBER), string='999', start=(1, 2), end=(1, 5), line='a=999'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

compared to in Python 3.7 on release candidate.

Version info sys.version_info(major=3, minor=7, micro=1, releaselevel='candidate', serial=1)
[TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a=999'),
 TokenInfo(type=53 (OP), string='=', start=(1, 1), end=(1, 2), line='a=999'),
 TokenInfo(type=2 (NUMBER), string='999', start=(1, 2), end=(1, 5), line='a=999'),
 TokenInfo(type=4 (NEWLINE), string='', start=(1, 5), end=(1, 6), line=''),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

adding a newline to the end of the statement will created a consistent tokenization.

Version info sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)
[TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a=999\n'),
 TokenInfo(type=53 (OP), string='=', start=(1, 1), end=(1, 2), line='a=999\n'),
 TokenInfo(type=2 (NUMBER), string='999', start=(1, 2), end=(1, 5), line='a=999\n'),
 TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='a=999\n'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 5, 2018

@Carreau it looks like the core tests are passing, but I am receiving a timeout error in the terminal test group . Do you have an insight into this?

@tonyfast @bollwyvl
Uncomment the ! assertion.
c11265e 
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 6, 2018

The errors at the moment are in SystemAssign. The test case =! is trying to assign a system statement.

tonyfast and others added 3 commits October 6, 2018 11:47
@tonyfast @bollwyvl
Add an extra condition to SystemAssign.
f7642d4 
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
@tonyfast @bollwyvl
Add check_complete test for exit.
76599ae 
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
@tonyfast @bollwyvl
Remove a condition from check_complete
bd7a4c2 
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
@tonyfast
Copy link
Contributor Author

tonyfast commented Oct 6, 2018

Woo! The new and old tests are passing now. These fixes resolve #11334 #11354 #11346 #11337 .

@Carreau @takluyver could you review this when you get a chance?

@Carreau
Copy link
Member

Carreau commented Oct 6, 2018

Thanks ! Awesome work @tonyfast !

I'll try to review and merced this week-end.

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
.travis.yml
@@ -34,6 +34,7 @@ after_success:

matrix:
include:
- { python: "3.6-dev", dist: xenial, sudo: true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just going to add some "ok", to follow where I am in the review for myself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok.

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
IPython/core/inputtransformer2.py
@@ -234,6 +234,7 @@ def find(cls, tokens_by_line):
for line in tokens_by_line:
assign_ix = _find_assign_op(line)
if (assign_ix is not None) \
and not line[assign_ix].line.strip().startswith('=') \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
IPython/core/inputtransformer2.py
@@ -369,11 +370,15 @@ def transform(self, lines):
end_line = find_end_of_continued_line(lines, start_line)
line = assemble_continued_line(lines, (start_line, start_col), end_line)

if line[:2] in ESCAPE_DOUBLES:
if len(line) > 1 and line[:2] in ESCAPE_DOUBLES:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
IPython/core/inputtransformer2.py
if escape in tr:
call = tr[escape](content)
else:
call = ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
IPython/core/inputtransformer2.py
# Append an newline for consistent tokenization
# See https://bugs.python.org/issue33899
cell += '\n'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seem good. Looks like we should either ends with 2 \n or None.

Carreau
Carreau reviewed Oct 6, 2018
View reviewed changes
IPython/core/inputtransformer2.py
if not lines:
return 'complete', None

if lines[-1].endswith('\\'):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

bxsx
bxsx reviewed Oct 6, 2018
View reviewed changes
IPython/core/tests/test_inputtransformer2.py Outdated
@@ -212,6 +214,17 @@ def test_check_complete():
nt.assert_equal(cc("a = '''\n hi"), ('incomplete', 3))
nt.assert_equal(cc("def a():\n x=1\n global x"), ('invalid', None))
nt.assert_equal(cc("a \\ "), ('invalid', None)) # Nothing allowed after backslash
nt.assert_equal(cc("1\\\n+2"), ('complete', None))
nt.assert_equal(cc("1\\\n+2"), ('complete', None))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant test

@tonyfast
Remove redundant test
bdaec6f 
Nice catch @bartskowron
@Carreau Carreau added this to the 7.1 milestone Oct 8, 2018
@Carreau
Copy link
Member

Carreau commented Oct 8, 2018

Thanks all, let's try to get that in and have an IPython 7.1 by end of week.
That should match the imminent release of Python 3.7.1 that have the compatibility issues mentioned earlier.

@Carreau Carreau merged commit 8e29b01 into ipython:master Oct 8, 2018
@Carreau Carreau added the Hacktoberfest you want to participate to hacktoberfest ? Here is an easy issue. label Oct 13, 2018
This was referenced Oct 15, 2018
Closed
Closed
Closed
@mamrhein mamrhein mentioned this pull request Nov 10, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bxsx bxsx bxsx left review comments

@Carreau Carreau Carreau left review comments

@takluyver takluyver Awaiting requested review from takluyver

Assignees
No one assigned
Labels
Hacktoberfest you want to participate to hacktoberfest ? Here is an easy issue.
Projects
None yet
Milestone
7.1
Development

Successfully merging this pull request may close these issues.

Cannot make multi-line code blocks in ipython
3 participants
@tonyfast @Carreau @bxsx