"Wrong" invalid character identified

[aroberge]

BPO	44335
Nosy	@aroberge, @pablogsal, @miss-islington
PRs	bpo-44335: Fix a regression when identifying invalid characters in syntax errors #26589 [3.10] bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589) #26600 bpo-44335: Ensure the tokenizer doesn't go into Python with the error set #26608 [3.10] bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608) #26610

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-08.11:28:25.044>
created_at = <Date 2021-06-07.17:07:47.584>
labels = ['3.10']
title = '"Wrong" invalid character identified'
updated_at = <Date 2021-06-08.19:25:25.215>
user = 'https://github.com/aroberge'

bugs.python.org fields:

activity = <Date 2021-06-08.19:25:25.215>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-08.11:28:25.044>
closer = 'pablogsal'
components = []
creation = <Date 2021-06-07.17:07:47.584>
creator = 'aroberge'
dependencies = []
files = []
hgrepos = []
issue_num = 44335
keywords = ['patch']
message_count = 5.0
messages = ['395267', '395324', '395325', '395353', '395356']
nosy_count = 3.0
nosy_names = ['aroberge', 'pablogsal', 'miss-islington']
pr_nums = ['26589', '26600', '26608', '26610']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44335'
versions = ['Python 3.10']

[aroberge]

When using Python 3.10.0b2 on a line with more than one invalid characters, the second one is identified as being incorrect, whereas in previous versions the first such character was identified.

py -3.8 unicode_quote.py
File "unicode_quote.py", line 2
a = « hello » « world »
^
SyntaxError: invalid character in identifier

py -3.9 unicode_quote.py
File "C:\...\unicode_quote.py", line 2
a = « hello » « world »
^
SyntaxError: invalid character '«' (U+00AB)

py -3.10 unicode_quote.py
File "C:\...\unicode_quote.py", line 2
a = « hello » « world »
^
SyntaxError: invalid character '»' (U+00BB)

[pablogsal]

New changeset d334c73 by Pablo Galindo in branch 'main':
bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589)
d334c73

[miss-islington]

New changeset 933b5b6 by Miss Islington (bot) in branch '3.10':
bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589)
933b5b6

[pablogsal]

New changeset bafe0aa by Pablo Galindo in branch 'main':
bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608)
bafe0aa

[miss-islington]

New changeset 2a8d712 by Miss Islington (bot) in branch '3.10':
bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608)
2a8d712