locale.getdefaultlocale() missing corner case

[rg3]

BPO	5815
Nosy	@malemburg, @loewis, @birkenfeld, @vstinner, @bitdancer, @serhiy-storchaka
Files	locale.diff locale_parse.patch locale_parse_2.patch locale_parse_2a.patch: Patch without changes to locale_alias.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-12-19.20:34:17.063>
created_at = <Date 2009-04-22.18:20:44.211>
labels = ['type-bug', 'library', 'docs']
title = 'locale.getdefaultlocale() missing corner case'
updated_at = <Date 2013-12-19.20:34:17.062>
user = 'https://bugs.python.org/rg3'

bugs.python.org fields:

activity = <Date 2013-12-19.20:34:17.062>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-12-19.20:34:17.063>
closer = 'serhiy.storchaka'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2009-04-22.18:20:44.211>
creator = 'rg3'
dependencies = []
files = ['13737', '26380', '31740', '31742']
hgrepos = []
issue_num = 5815
keywords = ['patch']
message_count = 40.0
messages = ['86312', '86313', '86317', '86318', '86319', '86327', '86332', '164859', '165264', '165267', '165447', '165448', '192461', '197572', '197575', '197577', '197580', '197581', '197583', '197585', '197588', '197599', '202498', '202544', '202565', '202601', '202603', '202633', '202644', '202645', '202647', '202682', '206555', '206558', '206632', '206634', '206638', '206645', '206646', '206647']
nosy_count = 11.0
nosy_names = ['lemburg', 'loewis', 'georg.brandl', 'vstinner', 'r.david.murray', 'rg3', 'Dmitry.Jemerov', 'python-dev', 'serhiy.storchaka', 'groodt', 'mfabian']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5815'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[rg3]

A recent issue with one of my programs has shown that
locale.getdefaultlocale() does not handle correctly a corner case. The
issue URL is this one:

http://bitbucket.org/rg3/youtube-dl/issue/7/

Essentially, some users have LANG set to something like
es_CA.UTF-8@valencia. In that case, locale.getdefaultlocale() returns,
as the encoding, the string "utf_8_valencia", which cannot be used as an
argument to the string encode() function. The obvious correct encoding
in this case is UTF-8.

I have traced the problem and it seems that it could be fixed by the
attached patch. It checks if the encoding, at that point, contains the
'@' symbol and, in that case, removes everything starting at that point,
leaving only "UTF-8".

I am not sure if this patch or a similar one should be applied to other
Python versions. My system has Python 2.5.2 and that's what I have patched.

Explanation as to why I put the code there:

• The simple case, es_CA.UTF-8 goes through that point too and enters
  the "if".
• I wanted to remove what goes after the '@' symbol at that point, so it
  either needed to be removed before the call to the normalizing function
  or inside the normalization.
• As this is not what I would consider a normalization, I put the code
  before the function call.

Thanks for your hard work. I hope my patch is valid.

Regards.

[rg3]

I just realized that the "if" I introduced is not really needed.
"encoding = encoding.split('@')[0]" works whether the '@' symbol is
present or not.

[bitdancer]

I wasn't able to reproduce this by just setting my LC_ALL environment
variable to es_CA.UTF-8@valencia and calling getdefaultlocale. Can you
provide more complete steps to reproduce?

[rg3]

You are right. The issue is not reproduced with es_CA.UTF-8@valencia but
with ca_ES.UTF-8@valencia. The fact that the first case works makes me
think maybe there's another way to solve the problem. Can you check that?

[rg3]

Further investigation:

The guy who had this issue may be from Valencia, Spain. According to the
manpage for setlocale(3) in my system, the form is usually
language[_territory][.codeset][@modifier]. So, in this case, it would
make sense for the language to be "ca" (Catalan) and territory "ES" (Spain).

My patch may be fine after all. Because, if at that point the @modifier
is still present (I have seen code that removes it before that point),
you'd still want to remove it and keep only the "codeset", which is the
interesting part.

[bitdancer]

OK, it turns out that this is one of a class of known bugs of long
standing (see bpo-554676 and bpo-1080864, for example). The
recommended solution is to not use locale.getdefaultlocale, but to use
locale.getperferredencoding. I have confirmed that that works for the
case of ca_ES.UTF-8@valencia in python2.5.

There is at least a doc bug here, since no mention of this
fragility/recommendation is made in the getdefaultlocale documentation.

Using getpreferredencoding seems to be the correct solution to your
problem. However, the locale.py module contains a number of examples of
modifiers in the locale_alias table. Presumably this case could be
added, but it is not clear to me what the policy is on that at this
time, so I'm adding Martin to the nosy list looking for some guidance.

[rg3]

Excellent. Thanks for the tip. I'll now proceed to modify my code to use
getpreferredencoding. Still, I think getdefaultlocale should work
because it could be used in other situations, I suppose.

[groodt]

Bumping this as part of a bug scrub at EuroPython. Is this still an issue? Should we fix in docs or in code?