using Accept-Language in the default locale negotiator

[tisdall]

Is there a rational behind not using the Accept-Language values in a HTTP request as a fallback in the default_locale_negotiator? I understand the preference of using a _LOCALE_ value over the Accept-Language as there seems to be issues with people properly setting that value, but why not use that value when there's no _LOCALE_ set? I think a lot of browsers set that browser's language automatically on install based on the OS's settings so it typically defaults to a reasonable value.

I wanted to see if there was some reason for this before submitting any sort of pull request.

[stevepiercy]

FWIW, in another framework I use, the Accept-Language request header is used to determine a language. The priority of determining a language in the other framework is:

• configured in the application (e.g., user preferences)
• else try Accept-Language header
• else use default language in the application

I think your suggestion is a good idea, but I'd get at least one more contributor to comment on this.

[tisdall]

@stevepiercy Which framework is that?

One issue I'm seeing now is that Accept- header processing usually requires a knowledge of what's available to match against. So, the client could give a list like da, en-gb;q=0.8, en;q=0.7 and normally you'd then take a look at what languages you have and give a best match, however the language negotiator doesn't have any standardized way of knowing what languages are available. The best that could be done without major changes is to just return da in that case and ignore what possible languages are provided.

I think it'd be nice to standardize a way in the configuration to specify what languages are available for matching against, though.

[tisdall]

What about a format like the following:

pyramid.available_languages = 
    en English
    en-US English (United States)
    fr French
    pr-BR Portuguese (Brazil)

The settings would be split by line feed and then each line is split by the first whitespace. The rest of the line is plain human language to use for display in user's code (ex. a drop down to select a language to set the _LOCALE_).

[mcdonc]

I don't think there is a single set of "available languages, FWIW. Something like the above is described in http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/narr/i18n.html#detecting-available-languages but it's really a "per-application" setting.

[tisdall]

@mcdonc - yes, I know. What I'm saying is creating a single definition for use in the default language negotiator. Then the default language negotiator can be a little smarter about matching languages when reading the "Accept-Language" as it may list more than one option.

Ex.
I have English and French translations in my application. I get a string saying da, en-gb;q=0.8, en;q=0.7. Currently the default_language_negotiator has no notion of what languages are available so it can only return "da". If it had a list such as ['en', 'fr'] then it can appropriately return "en" to match that accept-language string. In order to create such a list, we need a well defined method of setting a list in the settings (that's what I'm putting forward).

[stevepiercy]

@tisdall the framework is Knop written in Lasso by a Swede who really knows his räksmörgås. Here's an example of its implementation.
https://github.com/knop-project/knop/blob/master/docs/knop_manual.md#knop_lang

The language object has a default language name using ISO-639 and ISO-3166 by convention. One can either dynamically insert language strings into a language object, or insert the language strings via a config file.

Anyway, if I understand @mcdonc and the docs correctly, there is no opinion but there is a suggestion of how to implement what you want. You could use that suggestion to split on lines first, then parse the line to get both the language code and its human-friendly name.

I think this would be at least a good pyramid_cookbook recipe. There are a couple of implementations through mako and chameleon already. I really don't know enough about Pyramid and this topic to have an opinion about its adoption into the core.

[tisdall]

@mcdonc @stevepiercy - I added a simple PR to better explain what I'm proposing

[ztane]

We did the language list approach exactly like that (available_languages in .ini); however there is a problem with the matching that for example Chinese locales have complicated names, and we needed to match everything that could possibly come from the client with whatever our locale names were and it wasn't eventually pretty at all.

[mmerickel]

This definitely belongs in a recipe instead of in core but it'd be really nice if someone could write up an example for the cookbook or even as part of the i18n chapter. For example update http://docs.pylonsproject.org/projects/pyramid/en/1.6-branch/narr/i18n.html#detecting-available-languages

with a snippet negotiator that used request.accept_language.best_match() as you're describing and then fallback to default_locale_negotiator if no match was found.

[ztane]

I do not have time now, but here is a starter for someone:

#: Mapping of language codes send by browsers to supported dialects.
BROWSER_LANGUAGES = {
    'en-ca': 'en_GB',
    'en-gb': 'en_GB',
    'en-us': 'en_US',
    'fi':    'fi_FI',
    'fi-fi': 'fi_FI',
    'en':    'en_GB',
    'zh':    'zh_CN',
    'zh-cn': 'zh_CN',
    'zh-hk': 'zh_HK',
    'zh-tw': 'zh_TW'
}


def locale_negotiator(request):
    """Locale negotiator for pyramid views.

This version differs from Pyramid's :py:func:`default_locale_negotiator
<pyramid.i18n.default_locale_negotiator>` in that it has a fallback to the
preferred languages from browser.
"""
    locale = default_locale_negotiator(request)
    if locale is None and request.accept_language:
        locale = request.accept_language.best_match(BROWSER_LANGUAGES.keys()).lower()

        # default to en_GB as it does not have silly units etc...
        locale = BROWSER_LANGUAGES.get(locale, 'en_GB')
        request._LOCALE_ = locale

    return locale

[mmerickel]

It's been a while but I think that adding support for pyramid.available_languages is a fine idea. The PR would need to support the current behavior when this setting is not configured and would need to come with appropriate docs and tests. If there's no one interested in implementing this I'll probably just close this issue.