tokenize: add tests for multi-line strings

[ericvsmith]

BPO	25422
Nosy	@ericvsmith
Files	issue-25422.diff issue-25422-1.diff issue-25422-2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2015-10-17.00:46:20.790>
created_at = <Date 2015-10-16.16:42:29.445>
labels = ['type-bug', 'library']
title = 'tokenize: add tests for multi-line strings'
updated_at = <Date 2015-10-17.00:46:20.789>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2015-10-17.00:46:20.789>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2015-10-17.00:46:20.790>
closer = 'eric.smith'
components = ['Library (Lib)']
creation = <Date 2015-10-16.16:42:29.445>
creator = 'eric.smith'
dependencies = []
files = ['40798', '40799', '40800']
hgrepos = []
issue_num = 25422
keywords = ['patch']
message_count = 4.0
messages = ['253075', '253106', '253107', '253108']
nosy_count = 2.0
nosy_names = ['eric.smith', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25422'
versions = ['Python 3.6']

[ericvsmith]

As part of fixing bpo-25311, I'm going to change how the string pattern recognition works. I want to make sure I don't break anything that currently works, but there are no tests for multi-line strings. I'll add those first, under this issue.

I'm not positive that the multi-line string behavior is correct or makes sense, but that's not the point: I just want to first add tests for the existing functionality to make sure I don't break it.

[ericvsmith]

Avoid truncating the expected results when the tokens are long.

[ericvsmith]

Removed the truncation of tokens. There's really no point to it, and it could be hiding bugs.

Also removed the truncation of token names, although none were ever truncated.

[python-dev]

New changeset 91c44dc35dfd by Eric V. Smith in branch 'default':
bpo-25422: Add tests for multi-line string tokenization. Also remove truncated tokens.
https://hg.python.org/cpython/rev/91c44dc35dfd