localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[stratakis]

BPO	31900
Nosy	@malemburg, @vstinner, @skrah, @serhiy-storchaka, @andreas-schwab, @stratakis
PRs	bpo-31900: Fix localeconv() encoding for LC_NUMERIC #4174 bpo-31900: Fix decimal for LC_NUMERIC != LC_CTYPE #5191 [3.6] bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174) #5192
Files	inconsistent_locale_encodings.py lc_numeric.py lc_numeric2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-28.16:53:58.445>
created_at = <Date 2017-10-30.13:41:05.693>
labels = ['3.8', '3.7', 'tests']
title = 'localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding'
updated_at = <Date 2018-11-28.16:53:58.444>
user = 'https://github.com/stratakis'

bugs.python.org fields:

activity = <Date 2018-11-28.16:53:58.444>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-28.16:53:58.445>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-10-30.13:41:05.693>
creator = 'cstratak'
dependencies = []
files = ['47246', '47377', '47386']
hgrepos = []
issue_num = 31900
keywords = ['patch']
message_count = 33.0
messages = ['305227', '305229', '305230', '305231', '305235', '305236', '305237', '307230', '308561', '309774', '309960', '309962', '309966', '309969', '309970', '309971', '309973', '309974', '309975', '309977', '309978', '309980', '309981', '309986', '309987', '309988', '309989', '309993', '310020', '310940', '327905', '330610', '330611']
nosy_count = 6.0
nosy_names = ['lemburg', 'vstinner', 'skrah', 'serhiy.storchaka', 'schwab', 'cstratak']
pr_nums = ['4174', '5191', '5192']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31900'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[stratakis]

Original bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1484497

It seems that on the development branch of Fedora, when we updated glibc from 2.26 to 2.26.90, test_float_with_comma started failing.

Details from the original bug report:

Under certain circumstances, when LC_NUMERIC is fr_FR.ISO8859-1 but LC_ALL is C.UTF-8, locale.localeconv() fails with
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: Invalid or incomplete multibyte or wide character

Apparently, the thousands separator (or something else) in the lconv is "\xa0" (unbreakable space in fr_FR.ISO8859-1), and it's being decoded with UTF-8.

This is tripped by Python's test suite, namely test_float.GeneralFloatCases.test_float_with_comma

[vstinner]

I can reproduce the bug with Python 3.6 on Fedora 26 and these locales:

• LC_ALL = LC_CTYPE = fr_FR (encoding = ISO8859-1)
• LC_NUMERIC= es_MX.utf8 (encoding = UTF-8)

Good: LC_NUMERIC = LC_CTYPE = LC_ALL = "es_MX.utf8"

haypo@selma$ env -i python3 -c 'import locale; locale.setlocale(locale.LC_ALL, "es_MX.utf8"); print(ascii(locale.localeconv()["thousands_sep"]))'

=> '\u2009'

Bug: LC_NUMERIC = "es_MX.utf8" but LC_CTYPE = LC_ALL = "fr_FR"

haypo@selma$ env -i python3 -c 'import locale; locale.setlocale(locale.LC_ALL, "fr_FR"); locale.setlocale(locale.LC_NUMERIC, "es_MX.utf8"); print(ascii(locale.localeconv()["thousands_sep"]))'

=> '\xe2\x80\x89'

[stratakis]

Tested the PR on a system with glibc 2.26.90 where the test was failing, and it successfully passed.

[serhiy-storchaka]

This is a duplicate of bpo-28604. See also bpo-25812.

[skrah]

Same as bpo-7442, I think.

[vstinner]

Oh wow, this bug is older than what I expected :-) I added support for non-ASCII thousands separator in 2012:

https://bugs.python.org/issue13706#msg151733

[vstinner]

inconsistent_locale_encodings.py of closed issue bpo-7442 is interesting: I copy it here.

[stratakis]

Pinging here. Is there some way I can help to move the issue forward?

[vstinner]

Oh. Another Python function is impacted by the bug, str.format:

$ env -i python3 -c 'import locale; locale.setlocale(locale.LC_ALL, "fr_FR"); locale.setlocale(locale.LC_NUMERIC, "es_MX.utf8"); print(ascii(f"{1000:n}"))'
'1\xe2\x80\x89000'

It should be '1\u2009000' ('1', '\u2009', '000').

[vstinner]

I completed my change. It now fixes locale.localeconv(), str.format() for int, float, complex and decimal.Decimal:

vstinner@apu$ ./python lc_numeric.py
LC_CTYPE: ('fr_FR', 'ISO8859-1')
LC_NUMERIC: ('es_MX', 'UTF-8')
decimal_point: '.'
thousands_sep: '\u2009'
grouping: [3, 3, 0]
int.__format__: '1\u2009234'
float.__format__: '1\u2009234'
complex.__format__: '1\u2009234+0j'
Decimal.__format__: '1\u2009234'