UTF-8: unpaired surrogates mishandled

[anonymous]

BPO	433882
Nosy	@malemburg, @loewis
Files	Utf8bugs.py: script explaining & demonstrating the bugs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/malemburg'
closed_at = <Date 2002-02-07.11:44:25.000>
created_at = <Date 2001-06-17.11:27:45.000>
labels = ['expert-unicode']
title = 'UTF-8: unpaired surrogates mishandled'
updated_at = <Date 2002-02-07.11:44:25.000>
user = 'https://bugs.python.org/anonymous'

bugs.python.org fields:

activity = <Date 2002-02-07.11:44:25.000>
actor = 'lemburg'
assignee = 'lemburg'
closed = True
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2001-06-17.11:27:45.000>
creator = 'anonymous'
dependencies = []
files = ['78']
hgrepos = []
issue_num = 433882
keywords = []
message_count = 5.0
messages = ['5090', '5091', '5092', '5093', '5094']
nosy_count = 2.0
nosy_names = ['lemburg', 'loewis']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue433882'
versions = []

[nobody]

Two bugs:

1. UTF-8 encoding of unpaired high surrogate produces
   an invalid UTF-8 byte sequence.

2. UTF-8 decoding of any unpaired surrogate produces
   an exception ("illegal encoding") instead of the
   corresponding 16-bit scalar value.

See attached file utf8bugs.py for example plus detailed
remarks.

[loewis]

Logged In: YES
user_id=21627

I think the codec should reject unpaired surrogates both
when encoding and when decoding. I don't have a copy of
ISO 10646, but Unicode 3.1 points out

# ISO/IEC 10646 does not allow mapping of unpaired
surrogates, nor U+FFFE and U+FFFF (but it does allow other
noncharacters).

So apparently, encoding unpaired surrogates as UTF-8 is
not allowed according to ISO 10646. I think Python should
follow this rule, instead of the Unicode one.

[malemburg]

Logged In: YES
user_id=38388

I'll look into this after I'm back from vacation on the 10.09.

[malemburg]

Logged In: YES
user_id=38388

I've checked in a patch which fixes bug 1 in the report.

I am unsure about "bug 2": I think that raising an exception is better than silently accepting bogus input data.

[malemburg]

Logged In: YES
user_id=38388

I fixed bug 2 as well. UTF-8 roundtrip safety is needed for Python (even for unpaired surrogates) since we use
UTF-8 as marshalling format for code objects, i.e. in PYC files.