unknown error handlers should be reported early

[hroncok]

BPO	37388
Nosy	@vstinner, @ezio-melotti, @methane, @serhiy-storchaka, @hroncok
PRs	bpo-37388: Check codec error handler in development mode #14341 bpo-37388: Add PyUnicode_Decode(str, 0) fast-path #14385 bpo-37388: Don't check encoding/errors during finalization #19409
Files	bench.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-26.00:13:23.084>
created_at = <Date 2019-06-24.14:10:52.129>
labels = ['3.9', 'expert-unicode']
title = 'unknown error handlers should be reported early'
updated_at = <Date 2020-04-07.14:07:50.672>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2020-04-07.14:07:50.672>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-26.00:13:23.084>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2019-06-24.14:10:52.129>
creator = 'hroncok'
dependencies = []
files = ['48435']
hgrepos = []
issue_num = 37388
keywords = ['patch']
message_count = 8.0
messages = ['346407', '346409', '346414', '346569', '346573', '346574', '346575', '365906']
nosy_count = 5.0
nosy_names = ['vstinner', 'ezio.melotti', 'methane', 'serhiy.storchaka', 'hroncok']
pr_nums = ['14341', '14385', '19409']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37388'
versions = ['Python 3.9']

[hroncok]

I was just bit by specifying an nonexisitng error handler for bytes.decode() without noticing.

Consider this code:

>>> 'a'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
'a'

Nobody notices that the error handler doesn't exist.

However:

>>> 'ž'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown error handler name 'Boom, Shaka Laka, Boom!'

The error is only noticeable once there is an error in the data.

While nobody could possibly mistake 'Boom, Shaka Laka, Boom!' for a valid error handler, I was bit by this:

>> b.decode('utf-8', errors='surrogate')

Which in fact should have been

>> b.decode('utf-8', errors='surrogateescape')

Yet I wasn't notified, because the bytes in question were actually decodeable as valid utf-8.

I suggest that unknown error handler should rise an exception immediately like this:

>>> 'b'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown error handler name 'Boom, Shaka Laka, Boom!'

[vstinner]

Getting an error handler is expensive compared to the time to encode text/decode bytes. Python loads the error handler lazily to provide best performances. Text codecs are performance critical for Python.

If we add a check, it should only be enabled in development (python3 -X dev) and/or debug mode (./configure --with-pydebug).

[vstinner]

bench.py: microbenchmark to measure the overhead of PR 14341.

[vstinner]

New changeset 22eb689 by Victor Stinner in branch 'master':
bpo-37388: Development mode check encoding and errors (GH-14341)
22eb689

[vstinner]

New changeset ed076ed by Victor Stinner in branch 'master':
bpo-37388: Add PyUnicode_Decode(str, 0) fast-path (GH-14385)
ed076ed

[vstinner]

I compared ref (commit e1a63c4) to patch (commit ed076ed). I ran bench.py using:

# edit Makefile.pre.in to use:
PROFILE_TASK=-m test.regrtest --pgo test_unicode test_codecs

./configure --enable-optimizations && make
./python -m venv env && env/bin/python -m pip install -U pyperf
./env/bin/python bench.py -o <file>.json

$ python3 -m pyperf compare_to ref_e1a63c4f2.json patch_ed076ed4.json 
0B: Mean +- std dev: [ref_e1a63c4f2] 76.7 ns +- 0.9 ns -> [patch_ed076ed4] 77.5 ns +- 2.9 ns: 1.01x slower (+1%)
1B: Mean +- std dev: [ref_e1a63c4f2] 92.9 ns +- 0.8 ns -> [patch_ed076ed4] 94.0 ns +- 2.4 ns: 1.01x slower (+1%)
5B: Mean +- std dev: [ref_e1a63c4f2] 106 ns +- 2 ns -> [patch_ed076ed4] 110 ns +- 2 ns: 1.04x slower (+4%)
10B: Mean +- std dev: [ref_e1a63c4f2] 105 ns +- 1 ns -> [patch_ed076ed4] 109 ns +- 1 ns: 1.03x slower (+3%)
25B: Mean +- std dev: [ref_e1a63c4f2] 108 ns +- 3 ns -> [patch_ed076ed4] 111 ns +- 3 ns: 1.03x slower (+3%)
100B: Mean +- std dev: [ref_e1a63c4f2] 114 ns +- 1 ns -> [patch_ed076ed4] 115 ns +- 2 ns: 1.01x slower (+1%)
1000B: Mean +- std dev: [ref_e1a63c4f2] 267 ns +- 3 ns -> [patch_ed076ed4] 253 ns +- 4 ns: 1.06x faster (-5%)

$ python3 -m pyperf compare_to ref_e1a63c4f2.json patch_ed076ed4.json  --table
+-----------+---------------+-----------------------------+
| Benchmark | ref_e1a63c4f2 | patch_ed076ed4              |
+===========+===============+=============================+
| 0B        | 76.7 ns       | 77.5 ns: 1.01x slower (+1%) |
+-----------+---------------+-----------------------------+
| 1B        | 92.9 ns       | 94.0 ns: 1.01x slower (+1%) |
+-----------+---------------+-----------------------------+
| 5B        | 106 ns        | 110 ns: 1.04x slower (+4%)  |
+-----------+---------------+-----------------------------+
| 10B       | 105 ns        | 109 ns: 1.03x slower (+3%)  |
+-----------+---------------+-----------------------------+
| 25B       | 108 ns        | 111 ns: 1.03x slower (+3%)  |
+-----------+---------------+-----------------------------+
| 100B      | 114 ns        | 115 ns: 1.01x slower (+1%)  |
+-----------+---------------+-----------------------------+
| 1000B     | 267 ns        | 253 ns: 1.06x faster (-5%)  |
+-----------+---------------+-----------------------------+

The overhead of my change is around 1 ns, 4 ns (on 106 ns) in the worst case (5B).

The change "looks" faster on the 1000B case, but it's likely a glitch of PGO compilation which is not really deterministic.

[vstinner]

Ok, the encoding and errors are now checked in almost all cases if you enable the development mode using -X dev command line option.

[vstinner]

New changeset d8acf0d by Victor Stinner in branch 'master':
bpo-37388: Don't check encoding/errors during finalization (GH-19409)
d8acf0d