decoding functions in _codecs module accept str arguments

[pitrou]

BPO	4874
Nosy	@malemburg, @amauryfa, @pitrou, @vstinner
Files	_codecs_bytes-2.patch mbdecode-unicode.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2009-01-22.12:25:56.020>
created_at = <Date 2009-01-07.23:48:15.674>
labels = ['extension-modules', 'type-bug', 'release-blocker']
title = 'decoding functions in _codecs module accept str arguments'
updated_at = <Date 2009-01-22.12:25:56.019>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2009-01-22.12:25:56.019>
actor = 'pitrou'
assignee = 'pitrou'
closed = True
closed_date = <Date 2009-01-22.12:25:56.020>
closer = 'pitrou'
components = ['Extension Modules']
creation = <Date 2009-01-07.23:48:15.674>
creator = 'pitrou'
dependencies = []
files = ['12770', '12831']
hgrepos = []
issue_num = 4874
keywords = ['patch']
message_count = 12.0
messages = ['79384', '79387', '79402', '79741', '79843', '79991', '80362', '80363', '80364', '80365', '80366', '80367']
nosy_count = 4.0
nosy_names = ['lemburg', 'amaury.forgeotdarc', 'pitrou', 'vstinner']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4874'
versions = ['Python 3.0', 'Python 3.1']

[pitrou]

The following function calls should raise a TypeError instead. Encoding
functions are fine (they only accept str).

>>> import codecs
>>> codecs.utf_8_decode('aa')
('aa', 2)
>>> codecs.utf_8_decode('éé')
('éé', 4)
>>> codecs.latin_1_decode('éé')
('éé', 4)

[vstinner]

Patch replacing "s*" parsing format by "y*" for:

• utf_7_decode()
• utf_8_decode()
• utf_16_decode()
• utf_16_le_decode()
• utf_16_be_decode()
• utf_16_ex_decode()
• utf_32_decode()
• utf_32_le_decode()
• utf_32_be_decode()
• utf_32_ex_decode()
• unicode_escape_decode()
• raw_unicode_escape_decode()
• latin_1_decode()
• ascii_decode()
• charmap_decode()
• mbcs_decode()

Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess
that there is not tests for all these functions :-/

Note: codecs documentation was already correct:

.. method:: Codec.decode(input[, errors])
(...)
*input* must be a bytes object or one which provides the read-only
character
buffer interface -- for example, buffer objects and memory mapped
files.

[malemburg]

On 2009-01-08 01:59, STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

Patch replacing "s*" parsing format by "y*" for:

• utf_7_decode()
• utf_8_decode()
• utf_16_decode()
• utf_16_le_decode()
• utf_16_be_decode()
• utf_16_ex_decode()
• utf_32_decode()
• utf_32_le_decode()
• utf_32_be_decode()
• utf_32_ex_decode()
• latin_1_decode()
• ascii_decode()
• charmap_decode()
• mbcs_decode()

These are fine.

• unicode_escape_decode()
• raw_unicode_escape_decode()

These changes are in line with their C API codec interfaces as well,
but those particular codecs could well also be made to work on Unicode
input, since unescaping can well be applied to Unicode as well.

I'll probably open a new item for this.

Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess
that there is not tests for all these functions :-/

The mbcs codec is only available on Windows.

All others are tested by test_codecs.py.

Which ones are skipped in your case ?

Note: codecs documentation was already correct:

.. method:: Codec.decode(input[, errors])
(...)
*input* must be a bytes object or one which provides the read-only
character
buffer interface -- for example, buffer objects and memory mapped
files.

That's not entirely correct: codecs are allowed to accept any
object type and can also return any object type. It up to them
to decide, e.g. a codec may accept both bytes and Unicode input
and always generate Unicode output when decoding.

I guess I have to review those doc changes...

[pitrou]

The patch is probably fine, but it would be nice to add some unit tests
for the new behaviour.

[malemburg]

On 2009-01-13 14:14, Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

The patch is probably fine, but it would be nice to add some unit tests
for the new behaviour.

+1 from my side.

Thanks for the patch, Viktor.

[vstinner]

New patch:

• Leave unicode_escape_decode() and raw_unicode_escape_decode()
  unchanged (still accept unicode string)
• Test changes (reject unicode for most codecs decode functions)
• Write tests for unicode_escape_decode() and
  raw_unicode_escape_decode() (there was no test for these functions)

[pitrou]

Committed in r68855, r68856. Thanks!

[amauryfa]

IMO, Modules/cjkcodecs/multibytecodec.c should be changed as well.

[pitrou]

Right, I hadn't thought of that.