tokenize yield an ERRORTOKEN if the identifier starts with a non-ascii char

[florentx]

BPO	9712
Nosy	@terryjreedy, @benjaminp, @florentx
Superseder	bpo-24194: Make tokenize recognize Other_ID_Start and Other_ID_Continue chars

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-08-30.14:41:43.763>
created_at = <Date 2010-08-30.07:42:55.735>
labels = ['type-bug']
title = 'tokenize yield an ERRORTOKEN if the identifier starts with a non-ascii char'
updated_at = <Date 2018-03-15.00:18:48.250>
user = 'https://github.com/florentx'

bugs.python.org fields:

activity = <Date 2018-03-15.00:18:48.250>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2010-08-30.14:41:43.763>
closer = 'benjamin.peterson'
components = []
creation = <Date 2010-08-30.07:42:55.735>
creator = 'flox'
dependencies = []
files = []
hgrepos = []
issue_num = 9712
keywords = []
message_count = 5.0
messages = ['115201', '115218', '240544', '313846', '313847']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'benjamin.peterson', 'flox', 'Joshua.Landau']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = '24194'
type = 'behavior'
url = 'https://bugs.python.org/issue9712'
versions = ['Python 3.1', 'Python 3.2', 'Python 3.4']

[florentx]

from io import BytesIO
from tokenize import tokenize, tok_name

sample = 'éléphants = "un éléphant, deux éléphants, ..."\nprint(éléphants)\n'
sampleb = sample.encode('utf-8')

exec(sample)
# output: un éléphant, deux éléphants, ...
exec(sampleb)
# output: un éléphant, deux éléphants, ...

module = BytesIO()
module.write(sampleb)
module.seek(0)

for line in tokenize(module.readline):
    print(tok_name[line.type], line)

# output:
ENCODING TokenInfo(type=57, string='utf-8', start=(0, 0), end=(0, 0), line='')
ERRORTOKEN TokenInfo(type=54, string='é', start=(1, 0), end=(1, 1), line='éléphants = "un éléphant, deux éléphants, ..."\n')
NAME TokenInfo(type=1, string='léphants', start=(1, 1), end=(1, 9), line='éléphants = "un éléphant, deux éléphants, ..."\n')
OP TokenInfo(type=53, string='=', start=(1, 10), end=(1, 11), line='éléphants = "un éléphant, deux éléphants, ..."\n')
STRING TokenInfo(type=3, string='"un éléphant, deux éléphants, ..."', start=(1, 12), end=(1, 46), line='éléphants = "un éléphant, deux éléphants, ..."\n')
NEWLINE TokenInfo(type=4, string='\n', start=(1, 46), end=(1, 47), line='éléphants = "un éléphant, deux éléphants, ..."\n')
NAME TokenInfo(type=1, string='print', start=(2, 0), end=(2, 5), line='print(éléphants)\n')
OP TokenInfo(type=53, string='(', start=(2, 5), end=(2, 6), line='print(éléphants)\n')
ERRORTOKEN TokenInfo(type=54, string='é', start=(2, 6), end=(2, 7), line='print(éléphants)\n')
NAME TokenInfo(type=1, string='léphants', start=(2, 7), end=(2, 15), line='print(éléphants)\n')
OP TokenInfo(type=53, string=')', start=(2, 15), end=(2, 16), line='print(éléphants)\n')
NEWLINE TokenInfo(type=4, string='\n', start=(2, 16), end=(2, 17), line='print(éléphants)\n')
ENDMARKER TokenInfo(type=0, string='', start=(3, 0), end=(3, 0), line='')

[benjaminp]

r84364

[JoshuaLandau]

This doesn't seem to be a complete fix; the regex used does not include Other_ID_Start or Other_ID_Continue from

https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

Hence tokenize does not accept '℘·'.

Credit to modchan from http://stackoverflow.com/a/29586366/1763356.

[terryjreedy]

Joshua opened bpo-24194 as a duplicate of this because he could not reopen this. I am leaving it open as the superseder for this as Serhiy has already added two dependencies there, and because this seems to be a duplicate in turn of bpo-1693050 (which I will close along with bpo-32987).

[terryjreedy]

Actually, bpo-1693050 and bpo-12731, about \w, are duplicates.