Fixes 11334/11354 Multi line code blocks #11358
Conversation
When codeop.compile_command() returns None it actually says "at least some part of the code was compiled successfully" which is not really important for checking if it's complete or not. Once we haven't got any errors during compilation process, we just want to check if there will be another nested block of code or not by checking a colon.
An assert statement needed to be commented out. I'm not sure what the statement was stating though.
Anyone have any ideas why this may be passing on everything, but Python 3.7dev? |
Maybe a change of API in generate_token or similar on Pyhton 3.7 dev. |
I'll scope it out. Thanks. |
AFAICT we need need to add a |
These changes are necessary for a failure on Python 3.7 dev https://bugs.python.org/issue33899
Thanks for these references @Carreau . I added some conditions for the newline tokens. Let's see if this helps. |
The CPython changes to tokenize effect 3.6 & 3.7. The I'm currently working on this pr on py37rc1. |
Likely, yes. That would be good. unless it make the tests take too long in
High case we can test only on one -dev
…On Thu, Oct 4, 2018, 06:29 Tony Fast ***@***.***> wrote:
The CPython changes to tokenize effect 3.6 & 3.7. The IPython tests are
not checking on Python3.6dev.
Should there be tests for 3.6dev because of this use case?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#11358 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAUez7XeLldVbRK8NZ01tJKXwkJuH2X1ks5uhg1IgaJpZM4XHPxz>
.
|
If we consider a simple statement like In Python 3.6 with the old tokenize:
compared to in Python 3.7 on release candidate.
adding a newline to the end of the statement will created a consistent tokenization.
|
@Carreau it looks like the core tests are passing, but I am receiving a timeout error in the terminal test group . Do you have an insight into this? |
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
The errors at the moment are in |
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>
Thanks ! Awesome work @tonyfast ! I'll try to review and merced this week-end. |
@@ -34,6 +34,7 @@ after_success: | |||
|
|||
matrix: | |||
include: | |||
- { python: "3.6-dev", dist: xenial, sudo: true } |
Carreau
Oct 6, 2018
Member
I'm just going to add some "ok", to follow where I am in the review for myself.
I'm just going to add some "ok", to follow where I am in the review for myself.
Carreau
Oct 6, 2018
Member
ok.
ok.
@@ -234,6 +234,7 @@ def find(cls, tokens_by_line): | |||
for line in tokens_by_line: | |||
assign_ix = _find_assign_op(line) | |||
if (assign_ix is not None) \ | |||
and not line[assign_ix].line.strip().startswith('=') \ |
Carreau
Oct 6, 2018
Member
ok
ok
@@ -369,11 +370,15 @@ def transform(self, lines): | |||
end_line = find_end_of_continued_line(lines, start_line) | |||
line = assemble_continued_line(lines, (start_line, start_col), end_line) | |||
|
|||
if line[:2] in ESCAPE_DOUBLES: | |||
if len(line) > 1 and line[:2] in ESCAPE_DOUBLES: |
Carreau
Oct 6, 2018
Member
ok
ok
if escape in tr: | ||
call = tr[escape](content) | ||
else: | ||
call = '' |
Carreau
Oct 6, 2018
Member
ok
ok
# Append an newline for consistent tokenization | ||
# See https://bugs.python.org/issue33899 | ||
cell += '\n' | ||
|
Carreau
Oct 6, 2018
Member
Seem good. Looks like we should either ends with 2 \n or None.
Seem good. Looks like we should either ends with 2 \n or None.
if not lines: | ||
return 'complete', None | ||
|
||
if lines[-1].endswith('\\'): |
Carreau
Oct 6, 2018
Member
ok
ok
@@ -212,6 +214,17 @@ def test_check_complete(): | |||
nt.assert_equal(cc("a = '''\n hi"), ('incomplete', 3)) | |||
nt.assert_equal(cc("def a():\n x=1\n global x"), ('invalid', None)) | |||
nt.assert_equal(cc("a \\ "), ('invalid', None)) # Nothing allowed after backslash | |||
nt.assert_equal(cc("1\\\n+2"), ('complete', None)) | |||
nt.assert_equal(cc("1\\\n+2"), ('complete', None)) |
bxsx
Oct 6, 2018
Contributor
redundant test
redundant test
Nice catch @bartskowron
Thanks all, let's try to get that in and have an IPython 7.1 by end of week. |
8e29b01
into
ipython:master
This pull request merges the recent input transformer changes from master. It builds off of the work by @bartskowron in #11354 to resolve #11334 . The changes pass tests locally, including the new tests provided by @Carreau; let's see what CI says.
@takluyver I commented out an assertion. I am not sure what the statement, is this change ok?