Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multilingual support #656

Closed
wants to merge 7 commits into from
Closed

Conversation

GeoSander
Copy link
Contributor

@GeoSander GeoSander commented Mar 5, 2021

Enables multilingual support for pygeoapi (core and plugins). Refer to issue #100 for discussion.

Work done

  • Server-independent language check (from request l parameter or Accept-Language header)
  • Make providers, processes and formatters language-aware
  • Add translate functionality for JSON language structs based on the current language setting (best match)
  • Support responses with Content-Language header
  • Multilingual metadata in configuration
  • Add documentation
  • Handle hreflang in links in configuration

I've decided to push the last item to a next PR. This PR is already big enough...
The generation/processing of all API links should be done in a separate function, imho (see "Upcoming PRs" below).

What does it do?

Users can now append a l=fr parameter to a pygeoapi URL and pygeoapi will return the requested page in French (for example). If the user connects to pygeoapi using a browser where the language has been set to French, the requested page will automatically be rendered in French (without the l=fr parameter), because the browser sent an Accept-Language header along with the request. However, this can always be overridden again using the l query parameter.

Remarks

  • This PR is quite large and mainly impacts the API core (and not so much the plugins). Most of the language support stuff revolves around the new l10n (localization) module, which turns the requested language (e.g. "fr", "fr-CH" or "fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5" into a best matching Babel Locale object. These Locale objects integrate well with Jinja2 templates using the i18n extension;
  • The only text that is translatable at this moment, are the text strings (names, titles, descriptions, etc.) in the server configuration. The pygeoapi-config.yaml now contains some example language structs for US English and Canadian French (more translations coming up);
  • In order to efficiently pass on language settings, I decided to make all API method calls more consistent, so api.py has undergone quite a lot of refactoring. All methods that require Request object data are now decorated by the @pre_process function, which converts the incoming Flask/Starlette Request into a (newly introduced) APIRequest instance. This class processes headers and query parameters to extract the required properties for the API methods. This results in less boilerplate and cleaner API method signatures, while still being explicit;
  • Developers of third-party providers and other plugins should add a requested_locale=None argument to the __init__ method of their plugin, and pass the requested_locale to the super().__init__() call as a second argument. This is all that needs to be done to make the plugin work again and to make it language-aware. However, it is up to the developer to implement the logic to query/process data and return it in the requested language. Please refer to the provided Sphinx documentation for more info.

Follow-ups

This PR provides the foundation to make pygeoapi language-aware. However, for full language support, there are still some things that need to be done.

Upcoming PRs should address:

  • Links generated by pygeoapi: the links underneath each collection item (for example) should become language-aware, i.e. internal link URLs should be appended with a l=<language> query parameter if the current page also was requested using that parameter (note: the add_locale function in the l10n module can be used for that.
    Internal links should also set the hreflang property to the user-specified language. For external links, this only applies if the server speaks that language (hard to figure out).
  • HTML templates: all hard-coded translatable strings in the templates need to be stored in separate files and processed by pybabel. so they can be managed by a translation tool (e.g. Transifex);
  • The Jinja2 i18n extension needs to be configured and the HTML templates need to be modified so that Jinja2 and Babel/gettext can inject the translations;
  • All core plugins (e.g. providers) are now language-aware, but the ones that can support multiple languages (i.e. the plugin backend supports it) should still implement functionality for it. The l10n module provides the necessary tools to do so;
  • Other suggestions...?

Some useful links concerning Jinja2/Babel/Transifex:

@GeoSander GeoSander force-pushed the multilingual branch 2 times, most recently from 75a8482 to d36895c Compare March 5, 2021 22:36
@GeoSander GeoSander marked this pull request as ready for review March 15, 2021 14:54
@GeoSander GeoSander changed the title Multilingual support [WIP] Multilingual support Mar 15, 2021
GeoSander added a commit to GeoSander/geocore-pygeoapi that referenced this pull request Mar 15, 2021
@GeoSander GeoSander force-pushed the multilingual branch 5 times, most recently from 874ee4f to d468835 Compare March 17, 2021 14:10
* All routed API methods are now decorated by @pre_process (consistency) and no longer have a headers+format argument but a request argument (**kwargs also removed)
* The pre_process decorator turns an incoming Flask/Starlette request into a generic APIRequest instance
* The new APIRequest class extracts all relevant info (params, data, locale, etc.) from the request and exposes them as properties
* Removed a lot of boilerplate (i.e. format checking) and wrapped that into methods
* Updated server-specific API calls in each route method (pass entire request object, not headers and query params)
* Updated OpenAPI page with "l" query param
* Added example translations (metadata)
* Changed plugin signature: added explicit locale attribute (instead of **kwargs)
* Moved locale processing to get_plugin_locale() function in l10n module
* API should pass raw requested locale to plugins, locale should always be set
* Fixed API tests and added APIRequest tests
* Prepared utils.py for Jinja2 i18n extension
* Rebased on commit b40297a and fixed compatibility with geopython#661 and geopython#662
@GeoSander
Copy link
Contributor Author

GeoSander commented Mar 18, 2021

Note: build should actually work, but 1/2 failed because of:

image

* Fixed EDR provider signature (added locale)
* Fixed EDR API routes and query function (and improved parameter-name handling)
* Fixed EDR tests
* Added new translate_dict function to l10n module (+ tests)
* Updated all render_j2_template calls with locale parameter
* Updated pygeoapi-test-config.yml with some language structs
* support both 'language' and 'languages' property in server config and provider definitions
* renamed and modified translate_dict() to more generic translate_struct() function (l10n module)
* remove Content-Language header from provider responses if provider has no language support and format is json(ld)
* updated tests
@jodygarnett
Copy link
Contributor

Note: build should actually work, but 1/2 failed because of:

You have reached your pull rate limit

I note that the GEOS and PostGIS projects are using osgeo docker repository https://repo.osgeo.org/#browse/search/docker for test images (no pull rate limit there).

@tomkralidis
Copy link
Member

Closing in lieu of #664

@tomkralidis tomkralidis closed this Jun 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants