tokenize doesn't handle __future__.unicode_literals correctly

[tiran]

BPO	2495
Nosy	@amauryfa, @tiran
PRs	bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib #6572

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-03-27.23:24:21.500>
created_at = <Date 2008-03-27.11:50:02.935>
labels = ['library']
title = "tokenize doesn't handle __future__.unicode_literals correctly"
updated_at = <Date 2018-04-23.01:09:05.618>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2018-04-23.01:09:05.618>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2008-03-27.23:24:21.500>
closer = 'amaury.forgeotdarc'
components = ['Library (Lib)']
creation = <Date 2008-03-27.11:50:02.935>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 2495
keywords = []
message_count = 2.0
messages = ['64582', '64609']
nosy_count = 2.0
nosy_names = ['amaury.forgeotdarc', 'christian.heimes']
pr_nums = ['6572']
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue2495'
versions = ['Python 2.6']

[tiran]

See r61976

Clear the blacklist and run the test with

./python Lib/test/regrtest.py -uall test_tokenize

to reproduce the issue.

[amauryfa]

Actually, the problem is that untokenize does not put spaces between two
consecutive string literals:

'' '' => ''''

Corrected with r61979.
Will backport