locale.py doesn't recognize valid locale setting

[childsplay]

BPO	1080864
Nosy	@malemburg, @loewis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/malemburg'
closed_at = <Date 2004-12-16.12:33:54.000>
created_at = <Date 2004-12-07.20:23:46.000>
labels = ['library']
title = "locale.py doesn't recognize valid locale setting"
updated_at = <Date 2004-12-16.12:33:54.000>
user = 'https://bugs.python.org/childsplay'

bugs.python.org fields:

activity = <Date 2004-12-16.12:33:54.000>
actor = 'childsplay'
assignee = 'lemburg'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2004-12-07.20:23:46.000>
creator = 'childsplay'
dependencies = []
files = []
hgrepos = []
issue_num = 1080864
keywords = []
message_count = 20.0
messages = ['23584', '23585', '23586', '23587', '23588', '23589', '23590', '23591', '23592', '23593', '23594', '23595', '23596', '23597', '23598', '23599', '23600', '23601', '23602', '23603']
nosy_count = 4.0
nosy_names = ['lemburg', 'loewis', 'zgoda', 'childsplay']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1080864'
versions = ['Python 2.3']

[childsplay]

stas@mobi:~$ locale
LANG=nb_NO
[...]

stas@mobi:~$ python
Python 2.3.4 (#2, Sep 24 2004, 08:39:09) 
[GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2
Type "help", "copyright", "credits" or "license" for
more information.
>>> import locale
>>> locale.getdefaultlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/locale.py", line 346, in
getdefaultlocale
    return _parse_localename(localename)
  File "/usr/lib/python2.3/locale.py", line 280, in
_parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: nb_NO
>>>

[loewis]

Logged In: YES
user_id=21627

Why do you want to use getdefaultlocale()?

[zgoda]

Logged In: YES
user_id=92222

getdefaultlocale() is often used to get default encoding for
current system locale.
And, if function is provided in standard library, why
shouldn't one use it?

[loewis]

Logged In: YES
user_id=21627

To get the default encoding for the current locale, you
should use locale.getpreferredencoding(). You should not use
getdefaultlocale becaus it is (IMO) inherently broken, and
should not have been part of the standard library in the
first place.

[malemburg]

Logged In: YES
user_id=38388

Of course, I don't agree with you, Martin :-)
locale.getdefaultlocale() does server a purpose, namely that
of getting the default locale setting. The encoding
information is an often used extension when setting the
locale in the OS environment. If not set, the module
provides common defaults.

The locale "nb_NO" is not known to the module alias table.
Which locale, language and encoding would that be ?

[loewis]

Logged In: YES
user_id=21627

There is no "default locale setting" in most operating
systems. In what sense is the value of the LANG environment
variable a "default"?

[malemburg]

Logged In: YES
user_id=38388

The LANG (and other similar OS environment variables) define
what the user wishes to see being used as locale in the
applications that are started in that environment. See the
setlocale man page for details.

On some OSes such as Windows these settings are stored
differently, which is why the locale module has provisions
for finding these settings (thanks to Fredrik).

[childsplay]

Logged In: YES
user_id=638376

The reason I use getdefaultlocale(), is to get a platform
independant way of getting the systems locale setting.
The biggest advantage is that on Windows a "Linux like"
locale is returnt so that I can use the same language
support stuff on all
platforms. (Win,Linux,OSX).
Besides, what's the problem of adding the missing locale?

[malemburg]

Logged In: YES
user_id=38388

Please provide some authorative source which describes the
locale your are using (nb_NO) and the commonly used encoding
for that locale (see the existing dictionary in locale.py).

[childsplay]

Logged In: YES
user_id=638376

This is what I've put into /python2.3/locale.py:

locale_alias = {....
    .......
    'bokmål':			'nb_NO.ISO8859-1',
    'nb':			  'nb_NO.ISO8859-1',
    'nb_no':			'nb_NO.ISO8859-1',
    'nynorsk':			'nn_NO.ISO8859-1',
    'nn':			  'nn_NO.ISO8859-1',
    'nn_no':		        'nn_NO.ISO8859-1',
   ....
   ....
}

I have tested it on a number of apps and it fixes the problem.

[loewis]

Logged In: YES
user_id=21627

MAL: that doesn't answer my question, though: In what sense
does getdefaultlocale get a "default" locale? default as
opposed to what custom setting?

childsplay: the problem with adding additional aliases is
a) we can never hope to get a complete list of locales, so
this is a never-ending maintenance problem, and
b) the dictionary might be wrong on some systems. E.g.
sometimes, 'nb' might denote an ISO-8859-15 locale, or a
UTF-8 locale (e.g. when UTF-8 becomes the standard encoding
on Unix some day). If so, Python will silently compute an
incorrect default - in particular wrt. the encoding of the
"default" locale.

[childsplay]

Logged In: YES
user_id=638376

I agree that "default" would probably be called "preferred".

@loewis:
a)
I agree with your point of view but I as a developer I just
want to get the current locale in use and locale.py serves
that purpose in a platform independant way. The
"never-ending maintenance problem" is the result of the
locale horror we all have to live with until there's a final
solution/standard for it.

b)
Agree, I now understand your "getdefaultlocale ..
inherently broken" comment. But we still have a locale
module that supports some of the valid locales but not all,
which is (IMO) worse then having none at all.

BTW: getting the current locale to get a platform
independant language setting, should perhaps be part of
gettext.py instead of locale.py?

[malemburg]

Logged In: YES
user_id=38388

Martin:

"default" as opposed to whatever locale setting is currently
active for the program, i.e. the locale setting the program
would see after a single call to setlocale(LC_ALL, "") right
after the start of the program.

getdefaultlocale() mimics the lookup mechanism of
setlocale(LC_ALL, "").

The fact that the alias table may sometimes not give the
correct encoding is not a fault of Python or the table - if
the user wants to see a different encoding used as default
encoding for the set locale, then the user should include
that information in the LANG (or other) OS environment
variable of the process running the Python program.

Note that this is different than the "preferred" encoding
which a user can set in a window manager (KDE or Gnome) or
browser. Those settings are restricted to certain
application spaces. getdefaultencoding() is targetted at the
OS level setting which may be different from e.g. a KDE
setting (think a program running in a shell vs. a KDE
application run by a user).

[malemburg]

Logged In: YES
user_id=38388

childsplay (I wish people would use real names on SF...):

We can add the aliases you gave below, but we need some URLs
to add as reference. Please provide this information, so
that we can document that the aliases are in common use and
why iso-8859-1 is their usually used encoding.

Thanks.

[loewis]

Logged In: YES
user_id=21627

getdefaultencoding might be "targeted" at the OS level - but
the implementation certainly is not. On the OS level, the C
library will often use a different encoding after
locale.setlocale(locale.LC_ALL,"") is called, compared to
what getdefaultencoding returns. The approach takien inside
getdefaultencoding is inherently flawed, and cannot possibly
work in all cases; getpreferredencoding fixes that flaw.

[malemburg]

Logged In: YES
user_id=38388

Well, if the alias mapping is good enough for X, then it's
good enough for me :-)

I think we ought to update the alias table with the current
data of the X locale.alias file. This file also includes the
mappings that childsplay mentioned.

There also seems to be a bug in the encoding alias table:
"utf" is no longer recognized by setlocale(). This should be
changed to "utf8".

I'll fix that and post an update here.

[malemburg]

Logged In: YES
user_id=38388

Checking in Lib/locale.py;
/cvsroot/python/python/dist/src/Lib/locale.py,v <-- locale.py
new revision: 1.29; previous revision: 1.28
done

[childsplay]

Logged In: YES
user_id=638376

I've looked at the CVS/locale.py but is the utf8 entry missing?
Don't know about the X alias mapping, but I suspect that
there should also be an utf8 entry also for nb_NO/nn_NO.
(Just as there is for no_NO)

[malemburg]

Logged In: YES
user_id=38388

Thanks. I've noticed that the C lib doesn't seem to like
"UTF-8" but works well with "UTF8" (no hyphen).

Checking in Lib/locale.py;
/cvsroot/python/python/dist/src/Lib/locale.py,v <-- locale.py
new revision: 1.30; previous revision: 1.29
done
Checking in Tools/i18n/makelocalealias.py;
/cvsroot/python/python/dist/src/Tools/i18n/makelocalealias.py,v
<-- makelocalealias.py
new revision: 1.2; previous revision: 1.1
done

Please check again with the updated version.

[childsplay]

Logged In: YES
user_id=638376

Checked and it works ok, thanks.