test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode #63656

Arfrever · 2013-10-30T23:33:42Z

BPO	19457
Nosy	@malemburg, @vstinner, @benjaminp, @ezio-melotti, @serhiy-storchaka
Files	issue19457.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-10-31.15:24:47.269>
created_at = <Date 2013-10-30.23:33:42.339>
labels = ['type-bug', 'tests']
title = 'test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode'
updated_at = <Date 2013-10-31.15:24:47.267>
user = 'https://bugs.python.org/Arfrever'

bugs.python.org fields:

activity = <Date 2013-10-31.15:24:47.267>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-10-31.15:24:47.269>
closer = 'serhiy.storchaka'
components = ['Tests']
creation = <Date 2013-10-30.23:33:42.339>
creator = 'Arfrever'
dependencies = []
files = ['32432']
hgrepos = []
issue_num = 19457
keywords = ['patch']
message_count = 7.0
messages = ['201786', '201803', '201804', '201813', '201814', '201821', '201826']
nosy_count = 7.0
nosy_names = ['lemburg', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'Arfrever', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19457'
versions = ['Python 2.7']

Arfrever · 2013-10-30T23:33:41Z

test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() fail with Python supporting wide unicode, when they have been loaded from *.pyc files (test_codeccallbacks.pyc, test_unicode.pyc).
(This bug can be reproduced when running make test, which runs test suite twice, firstly with *.pyc files initially absent.)

This bug is a regression in 2.7.6rc1. These tests are absent in 2.7.5. These tests were added in 719ee60fc5e2.

$ ./configure --enable-unicode=ucs4
...
$ make
...
$ LD_LIBRARY_PATH="$(pwd)" ./python Lib/test/regrtest.py -v test_codeccallbacks
...
$ LD_LIBRARY_PATH="$(pwd)" ./python Lib/test/regrtest.py -v test_codeccallbacks
== CPython 2.7.6rc1 (2.7:dd12639b82bf, Oct 30 2013, 23:53:21) [GCC 4.8.1]
==   Linux-3.11.6
==   /tmp/cpython/build/test_python_6715
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
test_codeccallbacks
test_backslashescape (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodbackslashreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodignoreexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodstrictexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodxmlcharrefreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badhandlerresults (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badlookupcall (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badregistercall (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_bug828737 (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_callbacks (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_charmapencode (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_decodehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_decodeunicodeinternal (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_decoding_callbacks (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_encodehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_longstrings (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_lookup (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_translatehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unencodablereplacement (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodedecodeerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodeencodeerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodetranslateerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_uninamereplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unknownhandler (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharnamereplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharrefreplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharrefreplace_with_surrogates (test.test_codeccallbacks.CodecCallbackTest) ... FAIL
test_xmlcharrefvalues (test.test_codeccallbacks.CodecCallbackTest) ... ok

======================================================================
FAIL: test_xmlcharrefreplace_with_surrogates (test.test_codeccallbacks.CodecCallbackTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_codeccallbacks.py", line 93, in test_xmlcharrefreplace_with_surrogates
    exp, msg='%r.encode(%r)' % (s, encoding))
AssertionError: u'\U0001f49d'.encode('ascii')

Ran 29 tests in 0.071s

FAILED (failures=1)
test test_codeccallbacks failed -- Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_codeccallbacks.py", line 93, in test_xmlcharrefreplace_with_surrogates
    exp, msg='%r.encode(%r)' % (s, encoding))
AssertionError: u'\U0001f49d'.encode('ascii')

1 test failed:
test_codeccallbacks
$ LD_LIBRARY_PATH="$(pwd)" ./python Lib/test/regrtest.py -v test_unicode
...
$ LD_LIBRARY_PATH="$(pwd)" ./python Lib/test/regrtest.py -v test_unicode
== CPython 2.7.6rc1 (2.7:dd12639b82bf, Oct 30 2013, 23:53:21) [GCC 4.8.1]
== Linux-3.11.6
== /tmp/cpython/build/test_python_7518
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
test_unicode
test___contains__ (test.test_unicode.UnicodeTest) ... ok
test__format__ (test.test_unicode.UnicodeTest) ... ok
test_bug1001011 (test.test_unicode.UnicodeTest) ... ok
test_capitalize (test.test_unicode.UnicodeTest) ... ok
test_center (test.test_unicode.UnicodeTest) ... ok
test_codecs (test.test_unicode.UnicodeTest) ... ok
test_codecs_charmap (test.test_unicode.UnicodeTest) ... ok
test_codecs_errors (test.test_unicode.UnicodeTest) ... ok
test_codecs_idna (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf7 (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf8 (test.test_unicode.UnicodeTest) ... ok
test_comparison (test.test_unicode.UnicodeTest) ... ok
test_concatenation (test.test_unicode.UnicodeTest) ... ok
test_constructor (test.test_unicode.UnicodeTest) ... ok
test_contains (test.test_unicode.UnicodeTest) ... ok
test_conversion (test.test_unicode.UnicodeTest) ... ok
test_count (test.test_unicode.UnicodeTest) ... ok
test_encode_decimal (test.test_unicode.UnicodeTest) ... ok
test_encode_decimal_with_surrogates (test.test_unicode.UnicodeTest) ... FAIL
test_endswith (test.test_unicode.UnicodeTest) ... ok
test_expandtabs (test.test_unicode.UnicodeTest) ... ok
test_expandtabs_overflows_gracefully (test.test_unicode.UnicodeTest) ... ok
test_extended_getslice (test.test_unicode.UnicodeTest) ... ok
test_find (test.test_unicode.UnicodeTest) ... ok
test_find_etc_raise_correct_error_messages (test.test_unicode.UnicodeTest) ... ok
test_floatformatting (test.test_unicode.UnicodeTest) ... ok
test_format (test.test_unicode.UnicodeTest) ... ok
test_format_auto_numbering (test.test_unicode.UnicodeTest) ... ok
test_format_float (test.test_unicode.UnicodeTest) ... ok
test_format_huge_item_number (test.test_unicode.UnicodeTest) ... ok
test_format_huge_precision (test.test_unicode.UnicodeTest) ... ok
test_format_huge_width (test.test_unicode.UnicodeTest) ... ok
test_format_subclass (test.test_unicode.UnicodeTest) ... ok
test_formatting (test.test_unicode.UnicodeTest) ... ok
test_formatting_huge_precision (test.test_unicode.UnicodeTest) ... ok
test_formatting_huge_width (test.test_unicode.UnicodeTest) ... ok
test_hash (test.test_unicode.UnicodeTest) ... ok
test_index (test.test_unicode.UnicodeTest) ... ok
test_inplace_rewrites (test.test_unicode.UnicodeTest) ... ok
test_isalnum (test.test_unicode.UnicodeTest) ... ok
test_isalnum_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isalpha (test.test_unicode.UnicodeTest) ... ok
test_isalpha_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isdecimal (test.test_unicode.UnicodeTest) ... ok
test_isdecimal_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isdigit (test.test_unicode.UnicodeTest) ... ok
test_isdigit_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_islower (test.test_unicode.UnicodeTest) ... ok
test_islower_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isnumeric (test.test_unicode.UnicodeTest) ... ok
test_isnumeric_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isspace (test.test_unicode.UnicodeTest) ... ok
test_isspace_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_issue8271 (test.test_unicode.UnicodeTest) ... ok
test_istitle (test.test_unicode.UnicodeTest) ... ok
test_istitle_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_isupper (test.test_unicode.UnicodeTest) ... ok
test_isupper_non_bmp (test.test_unicode.UnicodeTest) ... ok
test_join (test.test_unicode.UnicodeTest) ... ok
test_literals (test.test_unicode.UnicodeTest) ... ok
test_ljust (test.test_unicode.UnicodeTest) ... ok
test_lower (test.test_unicode.UnicodeTest) ... ok
test_mul (test.test_unicode.UnicodeTest) ... ok
test_none_arguments (test.test_unicode.UnicodeTest) ... ok
test_partition (test.test_unicode.UnicodeTest) ... ok
test_printing (test.test_unicode.UnicodeTest) ... ok
test_raiseMemError (test.test_unicode.UnicodeTest) ... ok
test_replace (test.test_unicode.UnicodeTest) ... ok
test_replace_overflow (test.test_unicode.UnicodeTest) ... ok
test_repr (test.test_unicode.UnicodeTest) ... ok
test_rfind (test.test_unicode.UnicodeTest) ... ok
test_rindex (test.test_unicode.UnicodeTest) ... ok
test_rjust (test.test_unicode.UnicodeTest) ... ok
test_rpartition (test.test_unicode.UnicodeTest) ... ok
test_rsplit (test.test_unicode.UnicodeTest) ... ok
test_slice (test.test_unicode.UnicodeTest) ... ok
test_split (test.test_unicode.UnicodeTest) ... ok
test_splitlines (test.test_unicode.UnicodeTest) ... ok
test_startswith (test.test_unicode.UnicodeTest) ... ok
test_startswith_endswith_errors (test.test_unicode.UnicodeTest) ... ok
test_strip (test.test_unicode.UnicodeTest) ... ok
test_subscript (test.test_unicode.UnicodeTest) ... ok
test_surrogates (test.test_unicode.UnicodeTest) ... ok
test_swapcase (test.test_unicode.UnicodeTest) ... ok
test_title (test.test_unicode.UnicodeTest) ... ok
test_translate (test.test_unicode.UnicodeTest) ... ok
test_ucs4 (test.test_unicode.UnicodeTest) ... ok
test_unicode_repr (test.test_unicode.UnicodeTest) ... ok
test_upper (test.test_unicode.UnicodeTest) ... ok
test_utf8_decode_invalid_sequences (test.test_unicode.UnicodeTest) ... ok
test_utf8_decode_valid_sequences (test.test_unicode.UnicodeTest) ... ok
test_zfill (test.test_unicode.UnicodeTest) ... ok

======================================================================
FAIL: test_encode_decimal_with_surrogates (test.test_unicode.UnicodeTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_unicode.py", line 1672, in test_encode_decimal_with_surrogates
    '123' + exp)
  File "/tmp/cpython/Lib/test/test_unicode.py", line 45, in assertEqual
    super(UnicodeTest, self).assertEqual(first, second, msg)
AssertionError: '123&#128157;' != '123&#55357;&#56477;'

Ran 92 tests in 16.002s

FAILED (failures=1)
test test_unicode failed -- Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_unicode.py", line 1672, in test_encode_decimal_with_surrogates
    '123' + exp)
  File "/tmp/cpython/Lib/test/test_unicode.py", line 45, in assertEqual
    super(UnicodeTest, self).assertEqual(first, second, msg)
AssertionError: '123&#128157;' != '123&#55357;&#56477;'

1 test failed:
test_unicode

serhiy-storchaka · 2013-10-31T11:28:38Z

This issue is not release blocker because it affects only testing.

Here is a patch which should fix tests.

vstinner · 2013-10-31T11:31:26Z

+ if u'\ud83d\udc9d' != u'\U0001f49d':

If would prefer a test on sys.maxunicode, something like:

   if sys.maxunicode == 0xffff:

Oh, I didn't remember that Python supports surrogate pairs, but not always. Support of non-BMP characters in Python 2 is ugly :-)

serhiy-storchaka · 2013-10-31T13:19:07Z

   if u'\\ud83d\\udc9d' != u'\\U0001f49d':
If would prefer a test on sys.maxunicode, something like:

if sys.maxunicode == 0xffff:

No. 1. The check is true only on wide build. 2. It depends on how test module was loaded, true if it loaded from .py-file and false if it loaded from .py[co]-file.

vstinner · 2013-10-31T13:27:34Z

It depends on how test module was loaded, true if it loaded from .py-file and false if it loaded from .py[co]-file.

I tested with Python compiled in narrow or wide build: len(u'\ud83d\udc9d') value changes depending if the file is compiled to PYC in narrow or wide build. Oh, I have a headache. I didn't remember that Python 2 was so much broken with non-BMP characters :-p

python-dev · 2013-10-31T15:06:49Z

New changeset 8d5df9602a72 by Serhiy Storchaka in branch '2.7':
Issue bpo-19457: Fixed xmlcharrefreplace tests on wide build when tests are
http://hg.python.org/cpython/rev/8d5df9602a72

serhiy-storchaka · 2013-10-31T15:24:47Z

Thank you Arfrever for your report. And please describe a problem in the body of the issue, not in its title.

Arfrever mannequin added the release-blocker label Oct 30, 2013

Arfrever mannequin assigned serhiy-storchaka Oct 30, 2013

serhiy-storchaka added tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error and removed release-blocker labels Oct 31, 2013

serhiy-storchaka closed this as completed Oct 31, 2013

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode #63656

test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode #63656

Arfrever mannequin commented Oct 30, 2013

Arfrever mannequin commented Oct 30, 2013

serhiy-storchaka commented Oct 31, 2013

vstinner commented Oct 31, 2013

serhiy-storchaka commented Oct 31, 2013

vstinner commented Oct 31, 2013

python-dev mannequin commented Oct 31, 2013

serhiy-storchaka commented Oct 31, 2013

test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode #63656

test.test_codeccallbacks.CodecCallbackTest.test_xmlcharrefreplace_with_surrogates() and test.test_unicode.UnicodeTest.test_encode_decimal_with_surrogates() loaded from *.pyc files fail with Python supporting wide unicode #63656

Comments

Arfrever mannequin commented Oct 30, 2013

Arfrever mannequin commented Oct 30, 2013

serhiy-storchaka commented Oct 31, 2013

vstinner commented Oct 31, 2013

serhiy-storchaka commented Oct 31, 2013

vstinner commented Oct 31, 2013

python-dev mannequin commented Oct 31, 2013

serhiy-storchaka commented Oct 31, 2013