Faster charmap decoding

[serhiy-storchaka]

BPO	14874
Nosy	@malemburg, @loewis, @pitrou, @vstinner, @ezio-melotti, @serhiy-storchaka
Files	decode_charmap.patch charmapdecodebench.py: Benchmark script bench-diff.py: Benchmark results comparator

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-06-16.20:54:32.468>
created_at = <Date 2012-05-21.22:19:29.666>
labels = ['interpreter-core', 'expert-unicode', 'performance']
title = 'Faster charmap decoding'
updated_at = <Date 2012-06-16.20:54:32.466>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2012-06-16.20:54:32.466>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2012-06-16.20:54:32.468>
closer = 'pitrou'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2012-05-21.22:19:29.666>
creator = 'serhiy.storchaka'
dependencies = []
files = ['25664', '25665', '25666']
hgrepos = []
issue_num = 14874
keywords = ['patch']
message_count = 3.0
messages = ['161301', '162989', '162990']
nosy_count = 7.0
nosy_names = ['lemburg', 'loewis', 'pitrou', 'vstinner', 'ezio.melotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue14874'
versions = ['Python 3.3']

[serhiy-storchaka]

Charmap decoders are not as important as UTF decoders, but are still widely used. In Python 3.3 with PEP-393 they slowed down 4x. The proposed patch restores the performance.

Optimized only the most common case, when the decoder is specified by the UCS2 table with length >= 256. Map-based decoders translated to table-based. UCS1 tables widened to UCS2 by adding 257th fake characters.

Benchmark results:

                         3.2           3.3(vanilla)  3.3(patched)

cp1251 'A'*10000 111 (+10%) 31 (+294%) 122
cp1251 '\xa0'*10000 111 (+8%) 29 (+314%) 120
cp1251 '\u0402'*10000 111 (+6%) 25 (+372%) 118

[python-dev]

New changeset 8f3a5308f50b by Antoine Pitrou in branch 'default':
Issue bpo-14874: Restore charmap decoding speed to pre-PEP 393 levels.
http://hg.python.org/cpython/rev/8f3a5308f50b

[pitrou]

Thank you for the patch! Now pushed to 3.3.