tokenize.py improvements

[amikrop]

BPO	3078
Nosy	@rhettinger, @amikrop
Files	tokenize.py.patch: The patch in "diff -urN" format.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-06-17.13:56:34.827>
created_at = <Date 2008-06-11.13:10:44.878>
labels = ['library', 'performance']
title = 'tokenize.py improvements'
updated_at = <Date 2008-06-17.13:56:34.788>
user = 'https://github.com/amikrop'

bugs.python.org fields:

activity = <Date 2008-06-17.13:56:34.788>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2008-06-17.13:56:34.827>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2008-06-11.13:10:44.878>
creator = 'Indy'
dependencies = []
files = ['10581']
hgrepos = []
issue_num = 3078
keywords = ['patch']
message_count = 3.0
messages = ['67965', '68320', '68325']
nosy_count = 2.0
nosy_names = ['rhettinger', 'Indy']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue3078'
versions = ['Python 2.5']

[amikrop]

Various tokenize.py performance boosts and code clarifications.

[amikrop]

So, will this patch be applied?

[rhettinger]

Sorry, I don't see any value in this kind of patch.

The line "contline = += line" is broken. The "+=" transformations and
else-clause eliminations trivially re-arrange code without any real
savings. The "while 1" to "while True" transformation should not be
done in Py2.x because the latter is much slower ("while True" requires
loading a global variable and a test; in contrast, "while 1" is
optimized to an unconditional jump.