test_local.TestEnUSCollection failures on Solaris 10

[tpn]

BPO	16258
Nosy	@loewis, @jcea, @vstinner, @tpn, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-10-17.02:19:55.577>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'test_local.TestEnUSCollection failures on Solaris 10'
updated_at = <Date 2017-06-20.14:54:41.733>
user = 'https://github.com/tpn'

bugs.python.org fields:

activity = <Date 2017-06-20.14:54:41.733>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2012-10-17.02:19:55.577>
creator = 'trent'
dependencies = []
files = []
hgrepos = []
issue_num = 16258
keywords = ['3.3regression']
message_count = 17.0
messages = ['173124', '173164', '173166', '173167', '173168', '173171', '173172', '173199', '289382', '296414', '296415', '296416', '296418', '296435', '296440', '296441', '296445']
nosy_count = 6.0
nosy_names = ['loewis', 'jcea', 'vstinner', 'trent', 'serhiy.storchaka', 'petriborg']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16258'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

Linked PRs

• gh-60462: Fix locale.strxfrm() on Solaris #138242
• [3.14] gh-60462: Fix locale.strxfrm() on Solaris (GH-138242) #138448
• [3.13] gh-60462: Fix locale.strxfrm() on Solaris (GH-138242) #138449

[tpn]

======================================================================
ERROR: test_strxfrm (test.test_locale.TestEnUSCollation)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/3.x.snakebite-solaris10-u10ga2-sparc64-1/build/Lib/test/test_locale.py", line 346, in test_strxfrm
    self.assertLess(locale.strxfrm('a'), locale.strxfrm('b'))
ValueError: character U+101010e is not in range [U+0000; U+10ffff]

======================================================================
ERROR: test_strxfrm_with_diacritic (test.test_locale.TestEnUSCollation)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/3.x.snakebite-solaris10-u10ga2-sparc64-1/build/Lib/test/test_locale.py", line 367, in test_strxfrm_with_diacritic
    self.assertLess(locale.strxfrm('à'), locale.strxfrm('b'))
ValueError: character U+101010e is not in range [U+0000; U+10ffff]

---

Haven't investigated yet.

[tpn]

With the caveat that I know absolutely nothing about locales, here's what I've been able to reduce the problem down to:

zinc (alias s11, Solaris 11 x64):
    >>> locale.setlocale(locale.LC_ALL, 'C')
    'C'
    >>> locale.strxfrm('a')
    'a'
    >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
    'en_US.UTF-8'
    >>> locale.strxfrm('a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: character U+10105a3 is not in range [U+0000; U+10ffff]
    >>> 

nitrogen (alias s10, Solaris 10 SPARC):

    >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
    'en_US.UTF-8'
    >>> locale.strxfrm('a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: character U+101010e is not in range [U+0000; U+10ffff]

Not sure how relevant it is, but on both those Solaris boxes, locale.LC_ALL returns 6, whereas on BSD and OS X it always seems to return 0.

[jcea]

I can reproduce this on my x86 Solaris 10 update 10.

[pitrou]

With the system Python on s10:

Python 2.6.8 (unknown, Apr 13 2012, 17:08:12) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.strxfrm('a')
'a'
>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.strxfrm('a')
'\x01\x01\x01\x0e\x01\x01\x01\x01\x01\x01\x01\x02\x01\x01\x0fi\x01\x01\x01\x01'
>>> locale.strxfrm('a').decode('utf-8')
u'\x01\x01\x01\x0e\x01\x01\x01\x01\x01\x01\x01\x02\x01\x01\x0fi\x01\x01\x01\x01'

The difference between Python 2 and Python 3 is that Python 3 uses wcsxfrm, not strxfrm. Apparently Solaris' wcsxfrm is some broken thing that returns the same thing as strxfrm, cast to a wchar_t *, hence the character U+101010e (corresponding to the '\x01\x01\x01\x0e' bytestring above).

[jcea]

BTW, this works in python 3.2:

x86, 32 bit python, Solaris 10 update 10:

"""
Python 3.2.3 (default, Apr 12 2012, 13:29:13) 
[GCC 4.7.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.strxfrm('a')
'���\U00010f69�'
"""

[pitrou]

It only works on Python 3.2 because PyUnicode_FromWideChar is more permissive, it seems. The first character in the wchar_t string returned by Solaris is still 0x101010e.

[pitrou]

(by the way, I also tried a memset() before calling wcsxfrm(): no change)

[vstinner]

Python 3.2 rejects characters outside the range U+0000-U+10ffff in
some operations, but not everywhere. I fixed Python 3.3 to be more
strict and always reject characters outside this range. I noticed the
Solaris issue with mbstowcs() on locale encodings different than
UTF-8: bpo-13441. I asked if it's more important to be strict on Unicode,
or if we need to handle the wcsxfrm() issue on python-dev:
http://mail.python.org/pipermail/python-dev/2011-December/114759.html

Stefan Krah answered: "Yes, if the cause is a broken mbstowcs() that
sounds good."
http://mail.python.org/pipermail/python-dev/2011-December/114781.html

I asked for help on OpenIndiana IRC channel, but nobody had a locale
encoding different than UTF-8. I didn't have access to a Solaris box,
so I chose to skip failing tests on Solaris.

My commit 2a2d0872d993 (and 7ffe3d304487) skips many locales to
workaround this issue in test__locale.

[serhiy-storchaka]

May be bpo-15954 is related to this issue. Is this issue still reproduced?

[petriborg]

I'm getting the same 2 errors in Python 3.4.6 on Solaris 11.

Comes up when you run 'gmake test' or

./python -W default -bb -E -W error::BytesWarning -m test -r -w -j 0 -v test_locale.py

[vstinner]

A solution for that would be to return the raw byte string or to return a list of integers, rather than an unicode string.

I don't think that locale.strxfrm() result is supposed to be displayed in a terminal, it should only be used to sort two strings, or to be used as a key function for list.sort() for example.

[vstinner]

Currently, the function is documented to return a string:
https://docs.python.org/dev/library/locale.html#locale.strxfrm
"Transforms a string to one that can be used in locale-aware comparisons."

The problem is that we don't have enough developers who care of Solaris/Illimios to fix these issues (propose patches).

test_locale is just *one* example. The curses module is broken for years on Solaris if I recall correctly...

[serhiy-storchaka]

It is possible to use the special "encoding" for transformed strings on platforms with broken wcsxfrm().

All codes < 0x10000 are not changed. Codes >= 0x10000 are encoded as a pair: 0x10000 + (code >> 16), code & 0xffff.