tokenize

[lkcl]

BPO	34428
Nosy	@terryjreedy, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-25.02:08:02.993>
created_at = <Date 2018-08-18.10:29:27.237>
labels = ['3.8', '3.7', 'library', 'type-crash']
title = 'tokenize'
updated_at = <Date 2018-08-25.08:56:59.335>
user = 'https://bugs.python.org/lkcl'

bugs.python.org fields:

activity = <Date 2018-08-25.08:56:59.335>
actor = 'lkcl'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-25.02:08:02.993>
closer = 'terry.reedy'
components = ['Library (Lib)']
creation = <Date 2018-08-18.10:29:27.237>
creator = 'lkcl'
dependencies = []
files = []
hgrepos = []
issue_num = 34428
keywords = []
message_count = 9.0
messages = ['323698', '323700', '323704', '323705', '323707', '323708', '323709', '324029', '324054']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'lkcl', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue34428'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[lkcl]

hhatto/autopep8#414

the following two lines of code are not parseable by tokenize.py:

co = re.compile(
            "\(")

the combination of:

• being split on two lines
• having a backslash inside quotes
• having a bracket inside quotes

is an edge-case that _tokenize cannot cope with.

[lkcl]

these two line also pass (do not throw an exception):

co = re.compile(
            r"\(")

the code that fails may be further reduced to the following:

(
"\(")

[lkcl]

python2.7 and 3.5 also has exact same issue.

[serhiy-storchaka]

I can't reproduce.

>>> import tokenize
>>> list(tokenize.generate_tokens(iter(['(\n', r'"\(")']).__next__))
[TokenInfo(type=53 (OP), string='(', start=(1, 0), end=(1, 1), line='(\n'), TokenInfo(type=56 (NL), string='\n', start=(1, 1), end=(1, 2), line='(\n'), TokenInfo(type=3 (STRING), string='"\\("', start=(2, 0), end=(2, 4), line='"\\(")'), TokenInfo(type=53 (OP), string=')', start=(2, 4), end=(2, 5), line='"\\(")'), TokenInfo(type=4 (NEWLINE), string='', start=(2, 5), end=(2, 6), line=''), TokenInfo(type=0 (ENDMARKER), string='', start=(3, 0), end=(3, 0), line='')]

Could you please provide a minimal script that reproduces your issue?

[lkcl]

regular expressions are not something i am familiar or comfortable
with (never have been: the patterns are too dense). however REMOVING
"Bracket" from the regular expression(s) for PseudoToken "fixes"
the problem.

some debug print statements dropped in at around line 640 of
tokenize.py show that the match on the "working" code
with r"\(") as input gives a start/end/spos/epos that is DIFFERENT
from when the same code is given just "\("

line 'r"\\(")\n'
pos 0 7 r <_sre.SRE_Match object; span=(0, 5), match='r"\\("'>
pseudo start/end 0 5 (2, 0) (2, 5)

vs

line '"\\(")\n'
pos 0 6 " <_sre.SRE_Match object; span=(0, 4), match='"\\("'>
pseudo start/end 0 4 (5, 0) (5, 4)

there *may* be a way to "fix" this by taking out the pattern
matching on Bracket and prioritising everything else.

        while pos < max:
            pseudomatch = _compile(PseudoToken).match(line, pos)
            print ("pos", pos, max, line[pos], pseudomatch)
            if pseudomatch:                                # scan for tokens
                start, end = pseudomatch.span(1)
                spos, epos, pos = (lnum, start), (lnum, end), end
                print ("pseudo start/end", start, end, spos, epos)
                if start == end:
                    continue

Bracket = '[][(){}]'
Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
# REMOVE Bracket
Funny = group(Operator, Special)

PlainToken = group(Number, Funny, String, Name)
Token = Ignore + PlainToken

# First (or only) line of ' or " string.
ContStr = group(StringPrefix + r"'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
                group("'", r'\\\r?\n'),
                StringPrefix + r'"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
                group('"', r'\\\r?\n'))
PseudoExtras = group(r'\\\r?\n|\Z', Comment, Triple)
PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)

[lkcl]

wtf??? neither can i!!!!

import io
import tokenize

text = r'''\
(
r"\(")

(
"\(")
'''

string_io = io.StringIO(text)
tokens = list(
    tokenize.generate_tokens(string_io.readline)
)

print (tokens)

works perfectly.

ok ahhhh i bet you it's something to do with how
string_io.readline works, or something to do with
the format of the text. give me a sec to triage it.

[lkcl]

ahh darn-it, autopep8 is passing in tokens line-by-line,
to be parsed one at a time.... oh and of course it's losing
state information that tokenizer critically relies on.

i *think* that's what's going on.... so it's highly unlikely
to be a python tokenize bug... can we wait to see what the
autopep8 developer says?

[terryjreedy]

Luke: If and when you have legal code that fails when run with purely with CPython, you can re-open or try an new issue.

[lkcl]

yep good call terry, not getting any response from the
autopep8 developer, and i believe it was down to a loop
where the text is being thrown line-by-line at tokenize
and it was losing critical state information. so...
not a bug in tokenize.