Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Test and patch errors with unknown languages #1562

Closed
wants to merge 2 commits into from

2 participants

@beck

Test for two edge cases:

If an issued occurred on the filesystem and the .po files are no longer accessible, instead of raising an IOError, fails with ambiguous:

AttributeError: 'NoneType' object has no attribute '_info'

Related:
https://code.djangoproject.com/ticket/18192
https://groups.google.com/forum/#!msg/django-users/tc0asF6iFBo/u3cBN0SlUl0J

Second case: if trying to implement a language that django does not recognize (eg Inuktitut) translations fail with same error. Ticket: https://code.djangoproject.com/ticket/21055

beck added some commits
@beck beck Test for i18n bugs when language not found
Test for two edge cases:

If an issued occurred on the filesystem and the .po files are no longer accessible, instead of raising an IOError, fails with ambiguous:

 > AttributeError: 'NoneType' object has no attribute '_info'

Second case: if trying to implement a language that django does not recognize (eg Inuktitut) translations fail with same error.

Related:
https://code.djangoproject.com/ticket/18192
https://groups.google.com/forum/#!msg/django-users/tc0asF6iFBo/u3cBN0SlUl0J
abed38a
@beck beck Fixed #21055 #18192 -- Fix errors with unknown languages 11fc39b
@kelvin-wong

I am using 1.5.4.
After I look through the code, I found that the problem is about it's probably caused by inconsistency.

When the translation is searched in globalpath which is the path that contains all languages that django supported.
https://github.com/django/django/blob/1.5.4/django/utils/translation/trans_real.py#L113
https://github.com/django/django/blob/1.5.4/django/utils/translation/trans_real.py#L127

But in the check_for_language utils function, the translation is searched in all path that could contain locale files.
https://github.com/django/django/blob/1.5.4/django/utils/translation/trans_real.py#L351

So even I check the language is available by using check_for_language, I still got the error.
Does it make sense?

@beck

I found that the problem is about it's probably caused by inconsistency

Which problem? The pull addresses two issues:

  1. ambiguous error with IOError (a real issue seen by users)
  2. unknown language (less of a concern but should still be fixed for completeness)

Does it make sense?

Not completely, a code example would for sure help. If this pull is incomplete, I'm willing to add a test & fix.

@kelvin-wong

I just suggested that the root cause of django cannot find the translation files may be django have found them in the wrong places.

In the check_for_language function, it find the translation files in both global path and the user-defined paths using all_locale_paths.

def all_locale_paths():
    """
    Returns a list of paths to user-provides languages files.
    """
    from django.conf import settings
    globalpath = os.path.join(
        os.path.dirname(upath(sys.modules[settings.__module__].__file__)), 'locale')
    return [globalpath] + list(settings.LOCALE_PATHS)

But when it really try to get the translation files, django only find the global path.
So even we have defined the locale paths in settings and using check_for_language to identify the presence of the language, we still get this IOError.

@beck

After reviewing this pull once more (it's been several months), the real issue it not about unknown languages but the masked IOError when parsing translation mo files.

trans_real. translation() is very cryptic is difficult to review when changing. Since I've taken the time to untake how it all works, am closing this pull in favor of a rewrite: #2237

@beck beck closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 6, 2013
  1. @beck

    Test for i18n bugs when language not found

    beck authored
    Test for two edge cases:
    
    If an issued occurred on the filesystem and the .po files are no longer accessible, instead of raising an IOError, fails with ambiguous:
    
     > AttributeError: 'NoneType' object has no attribute '_info'
    
    Second case: if trying to implement a language that django does not recognize (eg Inuktitut) translations fail with same error.
    
    Related:
    https://code.djangoproject.com/ticket/18192
    https://groups.google.com/forum/#!msg/django-users/tc0asF6iFBo/u3cBN0SlUl0J
  2. @beck
This page is out of date. Refresh to see the latest.
View
35 django/utils/translation/trans_real.py
@@ -17,7 +17,7 @@
from django.utils.safestring import mark_safe, SafeData
from django.utils import six
from django.utils.six import StringIO
-from django.utils.translation import TranslatorCommentWarning
+from django.utils.translation import TranslatorCommentWarning, get_language_info
# Translations are cached in a dictionary for every language+app tuple.
@@ -137,6 +137,20 @@ def _translation(path):
res = _translation(globalpath)
+ # When a globalpath translation is none, one of two possibilities:
+ # 1- An IOError was encountered while searching for the .po files
+ # 2- Trying to implement a language unknown to django (uncommon but
+ # possible). Use an empty DjangoTranslation to enable following
+ # merges of all the locale directories
+ if res is None:
+ try:
+ get_language_info(lang)
+ raise IOError(("Unable to find translation files for "
+ "language:%s at path:%s"), lang, globalpath)
+ except KeyError:
+ res = DjangoTranslation()
+ res._catalog = {}
+
# We want to ensure that, for example, "en-gb" and "en-us" don't share
# the same translation object (thus, merging en-us with a local update
# doesn't affect en-gb), even though they will both use the core "en"
@@ -149,10 +163,7 @@ def _translation(path):
def _merge(path):
t = _translation(path)
if t is not None:
- if res is None:
- return t
- else:
- res.merge(t)
+ res.merge(t)
return res
for appname in reversed(settings.INSTALLED_APPS):
@@ -166,13 +177,13 @@ def _merge(path):
if os.path.isdir(localepath):
res = _merge(localepath)
- if res is None:
- if fallback is not None:
- res = fallback
- else:
- return gettext_module.NullTranslations()
- _translations[lang] = res
- return res
+ if res._catalog:
+ _translations[lang] = res
+ return res
+ if fallback._catalog:
+ _translations[lang] = fallback
+ return fallback
+ return gettext_module.NullTranslations()
default_translation = _fetch(settings.LANGUAGE_CODE)
current_translation = _fetch(language, fallback=default_translation)
View
BIN  tests/i18n/other/locale/iu/LC_MESSAGES/django.mo
Binary file not shown
View
26 tests/i18n/other/locale/iu/LC_MESSAGES/django.po
@@ -0,0 +1,26 @@
+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
+# This file is distributed under the same license as the PACKAGE package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: PACKAGE VERSION\n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2013-09-06 05:55+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+
+msgid "hello"
+msgstr "ᐊᐃ"
+
+msgid "younger sibling"
+msgstr "ᓄᑲᖅ"
+
+msgid "older sibling"
+msgstr "ᐊᖓᔪᒃ"
View
37 tests/i18n/tests.py
@@ -3,6 +3,7 @@
import datetime
import decimal
+import gettext as gettext_module
from importlib import import_module
import os
import pickle
@@ -1262,3 +1263,39 @@ def test_specific_language_codes(self):
r.META = {'HTTP_ACCEPT_LANGUAGE': 'pt-pt,en-US;q=0.8,en;q=0.6,ru;q=0.4'}
lang = get_language_from_request(r)
self.assertEqual('pt-br', lang)
+
+class LanguageNotFoundTests(TransRealMixin, TestCase):
+
+ def setUp(self):
+ super(LanguageNotFoundTests, self).setUp()
+ self.gettext_find_builtin = gettext_module.find
+
+ def tearDown(self):
+ gettext_module.find = self.gettext_find_builtin
+ super(LanguageNotFoundTests, self).tearDown()
+
+ def mockGettextFind(self):
+ def mocked_gettext_find(*args, **kwargs):
+ return None
+ gettext_module.find = mocked_gettext_find
+
+ @override_settings(
+ LANGUAGES=(
+ ('iu', 'Inuktitut'),
+ ),
+ LANGUAGE_CODE='iu',
+ LOCALE_PATHS=extended_locale_paths,
+ )
+ def test_unfamilar_language(self):
+ activate('iu-ca')
+ deactivate()
+
+ @override_settings(LANGUAGE_CODE='fr-xx')
+ def test_language_fallback(self):
+ activate('iu')
+ self.assertEqual('fr-xx', get_language())
+
+ def test_failure_finding_po_files(self):
+ self.flush_caches()
+ self.mockGettextFind()
+ self.assertRaises(IOError, activate, 'en')
Something went wrong with that request. Please try again.