str.format and string.Formatter bug with French (and other) locale

[canuck7]

BPO	35432
Nosy	@vstinner, @ericvsmith
Superseder	bpo-33954: float.format('n') fails with _PyUnicode_CheckConsistency assertion error for locales with non-ascii thousands separator

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-04.23:31:03.884>
created_at = <Date 2018-12-06.21:54:23.960>
labels = ['interpreter-core', 'type-bug']
title = 'str.format and string.Formatter bug with French (and other) locale'
updated_at = <Date 2019-01-04.23:31:03.882>
user = 'https://bugs.python.org/canuck7'

bugs.python.org fields:

activity = <Date 2019-01-04.23:31:03.882>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-04.23:31:03.884>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-12-06.21:54:23.960>
creator = 'canuck7'
dependencies = []
files = []
hgrepos = []
issue_num = 35432
keywords = []
message_count = 3.0
messages = ['331254', '331259', '333023']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.smith', 'canuck7']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '33954'
type = 'behavior'
url = 'https://bugs.python.org/issue35432'
versions = ['Python 3.6']

[canuck7]

The short story: Small numbers are not displayed properly when using a French (language) locale or similar, and formatting output with str.format or string.Formatter(). The problem probably extends to other locales.

Long story:

---

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
'fr_CA.UTF-8'
>>> print('{:n}'.format(10))    # Garbled output

>>> print('{:n}'.format(10000)) # OK
10 000
>>> # Note: narrow non-break space used as thousands separator
... pass
>>> locale.format_string('%d', 10, grouping=True)      # OK
'10'
>>> locale.format_string('%d', 10123)                  # OK
'10123'
>>> locale.format_string('%d', 10123, grouping=True)   # OK thousands separator \u202f
'10\u202f123'
>>> import string
>>> print(string.Formatter().format('{:n}', 10))  # Same problem with Formatter
AB
>>> print(string.Formatter().format('{:n}', 10000))
10 000

locale aware functions implementing the {:n} formatting code, such as str.format and string.Formatter, generate garbled output with small numbers under a French locale.

However, locale.format_string('%d', numeric_value) produces valid strings. In other words, it's a workaround for the time being...

The problem seems to originate from a new version of Ubuntu: I ran the same program about 18 months ago and didn't notice any problem.

My 0.02 $ worth of analysis: the output from the str.locale function is some random and changing value with small numbers. The behavior is reminiscent of invalid memory reads in C functions, e.g., mismatch of parameter in function calls, or similar. The value is not consistent. It feels like format does not expect and deal properly with long Unicode characters as part of numbers. The space character is a NARROW NON-BREAK SPACE, in most Ubuntu French locales (and quite a few others) however.

The problem shows up in Python 3.6 and 3.7.

This might also be a security issue...

[ericvsmith]

Possibly related to bpo-33954?

[vstinner]

This bug is a duplicate of bpo-33954. Good news: it's now fixed in Python 3.6.8!

https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-release-candidate-1

"bpo-33954: For str.format(), float.__format__() and complex.__format__() methods for non-ASCII decimal point when using the “n” formatter."

Note: I fixed other bugs with special locales (mostly LC_NUMERIC and/or LC_MONETARY using a different encoding than the LC_CTYPE locale).