locale.resetlocale is broken

[syvere]

BPO	504219
Nosy	@malemburg, @tim-one, @loewis, @mhammond, @akuchling, @birkenfeld, @birkenfeld, @ncoghlan, @devdanzin
Dependencies	bpo-813449: locale.getdefaultlocale doesnt handle all locales gracefully

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2002-01-16.04:56:12.000>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'locale.resetlocale is broken'
updated_at = <Date 2021-12-23.23:26:47.402>
user = 'https://bugs.python.org/syvere'

bugs.python.org fields:

activity = <Date 2021-12-23.23:26:47.402>
actor = 'ajaksu2'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2002-01-16.04:56:12.000>
creator = 'syvere'
dependencies = ['813449']
files = []
hgrepos = []
issue_num = 504219
keywords = []
message_count = 13.0
messages = ['8774', '8775', '8776', '8777', '8778', '8779', '8780', '8781', '114172', '185023', '185041', '240667', '293297']
nosy_count = 11.0
nosy_names = ['lemburg', 'tim.peters', 'loewis', 'mhammond', 'akuchling', 'georg.brandl', 'georg.brandl', 'jlgijsbers', 'syvere', 'meonkeys', 'ncoghlan', 'ajaksu2']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue504219'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[syvere]

locale.setlocale doesn't recognize the the format that
locale.py uses to set the locale, ie. no_NO and
friends.

The only way I've succeeded in setting the locale on
Python 2.1 is to use the format described in the
Visual C++ C-runtime library docs for setlocale where
a more verbose format is used to set the locale.

[tim-one]

Logged In: YES
user_id=31435

Mark, know anything about this? I don't.

[loewis]

Logged In: YES
user_id=21627

Can you provide a detailed test case? AFAIK, no_NO is indeed
no supported locale name on Windows, and I don't think
Python aanywhere uses it without the application explicitly
saying so.

[syvere]

Logged In: YES
user_id=428736

Sorry, I forgot to mention the testcase I am using. The
test case that fails is the locale module itself, when
running it as a standalone application that is.

To be more specific:
File "d:\devtools\python21\lib\locale.py", line 384, in
resetlocale
_setlocale(category, _build_localename(getdefaultlocale
()))
locale.Error: locale setting not supported

And to clarify what input getdefaultlocale returns on my
machine:

>>> locale.getdefaultlocale()
('no_NO', 'cp1252')

and:
>>> locale._build_localename(locale.getdefaultlocale())
'no_NO.cp1252'

By the way, is this bug fixed in python 2.2?

[jlgijsbers]

Logged In: YES
user_id=469548

Still reproducible with Python 2.4:

Python 2.4b2 (#19, Nov  8 2004, 11:15:07)
[GCC 3.3.5 (Debian 1:3.3.5-2)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import locale
>>> locale.getdefaultlocale()
['en_US', 'utf']
>>> locale.resetlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/johannes/python/Lib/locale.py", line 389, in
resetlocale
    _setlocale(category, _build_localename(getdefaultlocale()))
locale.Error: unsupported locale setting

Note that if I run python with 'LANG=nl_NL python', the
error does not occur.

http://python.org/sf/813449 seems to be the same bug, BTW.

[birkenfeld]

Logged In: YES
user_id=1188172

As this is not Windows specific, setting Category to Library.

[meonkeys]

Logged In: YES
user_id=259388

I'm seeing this error on Fedora Core 4:

Python 2.4.1 (#1, May 16 2005, 15:19:29) 
[GCC 4.0.0 20050512 (Red Hat 4.0.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import locale; locale.getdefaultlocale()
('en_US', 'utf')
>>> locale.resetlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/locale.py", line 389, in resetlocale
    _setlocale(category, _build_localename(getdefaultlocale()))
locale.Error: unsupported locale setting

[birkenfeld]

Logged In: YES
user_id=849994

The same applies to other locales.

From bpo-813449 (a duplicate of this):
"""
This is a known limitation: getdefaultlocale should not be
used in new code.

If the intention is to compute the locale's encoding,
locale.getpreferredencoding should be used instead.
"""

[BreamoreBoy]

Still a problems on py3k. Set stage to needs patch as it's so easy to reproduce.

[BreamoreBoy]

Can we simply document that getpreferredencoding should be used instead as stated in msg8781 ? Should getdefaultlocale be earmarked for deprecation?

[malemburg]

Adding support for locales that are not recognized is easy and the locale parser could also learn about formats that it doesn't yet understand, so patches are welcome.

The main problem here is that setlocale() only understands a very limited set of locale names.

Please note that locale.getdefaultlocale() is not the same as locale.getpreferredencoding():

• getdefaultlocale() aims to find the default locale settings for a program which has not yet called setlocale(LC_ALL, "").

• getpreferredencoding() tries to make an educated guess at the encoding the user has setup for his/her environment. It works well on Windows, but on Unix, requires a call to setlocale() to implement the environment variable parsing, which is exactly what getdefaultlocale() tries to avoid.
  setlocale() is not thread-safe, so the approach taken by getpreferredencoding() can have unwanted side-effects.

[akuchling]

It doesn't seem to me that we've really deprecated getdefaultlocale() -- it's not documented as such, and MAL makes the good point that getdefaultlocale() is trying to avoid calling setlocale().

Perhaps this is just a documentation problem? _build_localename() just glues together the language code and the encoding, resulting in things like en_US.ISO8859-1, which turns out to not work. So maybe we should document that getdefaultlocale() will sometimes guess wrong.

[ncoghlan]

Trying to come up with a short LC_MONETARY example for PEP-538, I just ran into what seems to be a related problem with locale.resetlocale(), which is that it doesn't work properly for categories other than LC_CTYPE: locale.getdefaultlocale() doesn't let you say which category you're actually interested in, so even if you specific a category for resetlocale(), it's going to look at the LC_CTYPE setting, *NOT* the one for the category you're interested in.

So perhaps a suitable design change here would be to update resetlocale() to just pass an empty string (letting the underlying platform API call figure out the right default), rather than passing the result of locale.getdefaultlocale()?

[arhadthedev]

The issue is no longer reproducible (at least on Windows 10; can someone test it on Vista?):

>>> import locale
>>> locale.getlocale(locale.LC_NUMERIC)
(None, None)
>>> locale.getdefaultlocale()
<stdin>:1: DeprecationWarning: Use setlocale(), getpreferredencoding(False) and getlocale() instead
('ru_RU', 'cp1251')
>>> locale.setlocale(locale.LC_ALL, 'no_NO')
'no_NO'
>>> locale.getlocale(locale.LC_ALL)
('no_NO', 'ISO8859-1')
>>> locale.getlocale(locale.LC_NUMERIC)
('no_NO', 'ISO8859-1')

[erlend-aasland]

The issue is no longer reproducible (at least on Windows 10; can someone test it on Vista?):

Windows 10, Norwegian Nynorsk locale, Python 3.11:

H:\> py -3.11
Python 3.11.0b1 (main, May  7 2022, 22:58:47) [MSC v.1931 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
<stdin>:1: DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead
('nn_NO', 'cp1252')
>>> locale.resetlocale()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python311\Lib\locale.py", line 636, in resetlocale
_setlocale(category, _build_localename(getdefaultlocale()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
locale.Error: unsupported locale setting
>>>

[vstinner]

Oh, I forgot to deprecate resetlocale() when I deprecated getdefaultlocale(). I proposed #93196 to deprecate resetlocale(). See #90817 for the rationale.

[AA-Turner]

locale.resetlocale() is now deprecated from Python 3.11 and will be removed in 3.13 (#93196), closing this issue.

A

[vstinner]

You should use locale.setlocale(locale.LC_ALL, "") instead.

[alexcTropic]

How can we modify the default value now?