Internationalization & Localisation

[SerhiyRomanov]

First of all, thank you for excellent framework! I've started used it for one personal project and find it very useful. Also i saw roadmap of framework - it look nice. But there are some things which would be very helpful if they will have integrated with Stralette and unfortunately, i didn't find any mentions about.

Among of them is localization and internationalization. Are any plans to do this via Middleware or something else? Let's discuss how it can be implemented...)

[tomchristie]

First things to consider:

• Are you interested here in multi-language support, or more simply a single language?
• Do we currently have any English error phrases or textual information that we reflect in the framework itself, that could be internationalized. (I'd treat HTTP Status Phrases in plaintext responses as an allowable exception here.)
• Are there any existing standalone Python packages that provide lazily translated strings, and jinja template tags?

The Django middleware is a good place to look for what behaviour we might want from an internationalization middleware: https://docs.djangoproject.com/en/2.1/topics/i18n/translation/#how-django-discovers-language-preference

Essentially we'd want to set a request.locale, based on either:

• The incoming URL, eg /it/orgnaisations
• A locale stored in the user session.
• A locale stored in a cookie.
• The Accept-Language header.
• A language configured to be the default.

[alex-oleshkevich]

I am also interested in this topic.

Using request.locale to set translator language sounds good.

@tomchristie what are the plans about translator? would you like to go with own implementation or integrate one of the existing libraries (babel?)?

Also, from my experience, various translation catalog loaders are also a good idea.
Example: load catalogs from po,yaml files, from database (my customers like to manage translation via GUI).

To be more precise, my use cases are:

1. In templates: {{ 'Some text'|trans }}
2. In python: __('Some text')

[tomchristie]

• what are the plans about translator? - None in particular at the moment.
• "various translation catalog loaders are also a good idea" - Sounds like a good plan.

It's also worth noting that I don't think Starlette itself currently has any strings that we ought to mark as translatable. (The plaintext responses like 404 Not Found I would count as technical messages that aren't anyway intended to end up in front of end-users in a production setup.)

However it's possible that at some point it might have some default login views that could include translatable text, and it will certainly end up recommending some third party libs that'll have translatable text strings (https://github.com/encode/typesystem is upcoming shortly, and would also tie in with any ORM implementation)

We'll likely want:

• A middleware class that sets a locale based on Accept-Language / Cookie / ...
• Use task-locals to store the locale, so that it's accessible elsewhere without having to be explicitly passed.
• A recommendation on how to use that with translatable strings (Both in templates, and in JSON-serialized responses)

[alex-oleshkevich]

@tomchristie thanks for reply.
i think this should be reflected in the roadmap so the community would know what exactly expected in the scope of starlette, what is out of scope and, probably, some raw technical decisions (as one you gave above).

[gnat]

I've run into this need as well, and have added this option using the official Jinja i18n extension in the Starlette Jinja2Templates class.

Some good news: It's not a large change, and will give Starlette full internationalisation / localisation support through it's template system, in a very standard way.

I will be filing a pull request shortly, after I've finished the new documentation.

[gnat]

Also thought it'd be worth mentioning: No additional packages are required, as the extension comes with Jinja by default.

[gnat]

Update: So after some digging and double checking of my work, I've realised both gettext and Babel from within Jinja's i18n extension use a global language state, and therefore I believe are unsuitable for Starlette.

Even if the translations were pre-cached and installed/uninstalled very quickly, there would be unforseen side effects because there could only be 1 active translation within the runtime at once.

A future solution will probably need to have all translations loaded at once, and the current translation for one specific render to be passed into the template system.

[gnat]

So there's a very easy way to do this per-request using simple dictionaries, using a format of your choice, without any sort of extra jinja2 extension or gettext or babel:

1. Pre-load your language translations globally as dictionaries.
2. For each request, pass the detected language translation to jinja2 parameters. Example:

translate = get_translation("en")
return templates.TemplateResponse('home.html', {'request': request, 'translate': translate})

3. Reference the translation within the template: {{ translate['hello world'] }}

[Kludex]

Should we just recommend https://github.com/klen/asgi-babel on our docs, and close this?

[gnat]

Haven't had the opportunity to audit or test performance on that middleware, but at a glance, it looks like a fair choice for leveraging babel for translations.

[dalf]

What is the recommended way for the Jinja2 templates?

In a attempt to migrate an application from Flask to Starlette, I have created a new class that inherit from starlette.templating.Jinja2Templates to change the Jinja2 environment initialization and to install the gettext callables:

• https://jinja.palletsprojects.com/en/3.0.x/extensions/#jinja2.Environment.install_gettext_callables

        # possible configuration with extension
        env = jinja2.Environment(
            loader=loader,
            autoescape=True,
            trim_blocks=True,
            lstrip_blocks=True,
            auto_reload=False,
            extensions=[
                'jinja2.ext.loopcontrols',
                'jinja2.ext.i18n'
            ],
        )

Is there a better way?

[gnat]

@dalf jinja2's i18n extension is unusable for async python: #279 (comment)

[boamaod]

Update: So after some digging and double checking of my work, I've realised both gettext and Babel from within Jinja's i18n extension use a global language state, and therefore I believe are unsuitable for Starlette.

So although https://github.com/bigbag/starlette-i18n is functional, you cannot recommend it for production? Can you explain consequences of using jinja2's i18n and the recommended alternative a bit more, @gnat? Does https://github.com/klen/asgi-babel overcome these problems and how? If one needs to implement i18n, what are the practical considerations to start with?

[frankie567]

Regarding Jinja templates, another solution I imagined was to subclass the Jinja2Templates class and overload the TemplateResponse to pass and install the gettext translations object right before the rendering. It looks like this:

class LocaleJinja2Templates(Jinja2Templates):
    def _create_env(self, directory):
        env = super()._create_env(directory)
        env.add_extension("jinja2.ext.i18n")
        return env

    def LocaleTemplateResponse(
        self,
        name: str,
        context: dict,
        translations: gettext.NullTranslations,
        status_code: int = 200,
        headers: dict = None,
        media_type: str = None,
        background: BackgroundTasks = None,
    ):
        self.env.install_gettext_translations(translations, newstyle=True)  # type: ignore
        return super().TemplateResponse(
            name, context, status_code, headers, media_type, background
        )

templates = LocaleJinja2Templates(directory='templates')

Then in a route, we just have to pass the gettext translations when calling LocaleTemplateResponse:

async def homepage(request):
    return templates.LocaleTemplateResponse('index.html', {'request': request}, translations=request.scope["translations"])

The way we retrieve the translations object can be anything we want. Here, I supposed we have a middleware setting it in the scope of the request.

Maybe a bit more verbose and less "magic" than the traditional ways, but it has the benefit to be very explicit.

[tomchristie]

Similarly to #284, #293 I'd like to close this off for now.

Super interested in seeing good third party options and approaches here, but I don't think we're in a position to consider anything extra in Starlette itself at the moment.