lib2to3: support non-ASCII identifiers

[holymonson]

BPO	34515
Nosy	@benjaminp, @holymonson, @miss-islington
PRs	bpo-34515: lib2to3: support non-ASCII identifiers #8950 [3.7] closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) #9333

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-15.17:32:33.841>
created_at = <Date 2018-08-27.06:47:47.478>
labels = ['3.8', 'library']
title = 'lib2to3: support non-ASCII identifiers'
updated_at = <Date 2018-09-15.17:53:02.447>
user = 'https://github.com/holymonson'

bugs.python.org fields:

activity = <Date 2018-09-15.17:53:02.447>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-15.17:32:33.841>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2018-08-27.06:47:47.478>
creator = 'monson'
dependencies = []
files = []
hgrepos = []
issue_num = 34515
keywords = ['patch']
message_count = 3.0
messages = ['324148', '325451', '325452']
nosy_count = 3.0
nosy_names = ['benjamin.peterson', 'monson', 'miss-islington']
pr_nums = ['8950', '9333']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34515'
versions = ['Python 3.8']

[holymonson]

Python 3.0 introduces additional characters from outside the ASCII range (see PEP-3131). see https://docs.python.org/3/reference/lexical_analysis.html#identifiers

But lib2to3 can't tokenize them corretly.

$ echo '中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	ERRORTOKEN	'中'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

'中' should be tokenized as NAME instead of ERRORTOKEN.

[benjaminp]

New changeset 10a428b by Benjamin Peterson (Monson Shao) in branch 'master':
closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)
10a428b

[miss-islington]

New changeset 51dbae8 by Miss Islington (bot) in branch '3.7':
closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)
51dbae8