i18n revamp (WORK IN PROGRESS)

[amcgee]

THIS IS A WORK IN PROGRESS, I will continue fleshing it out before sharing for comment

I'm opening this issue to discuss an overhaul of the internationalization (i18n) system. I'll start by laying out the challenges we face with the current system, and this issue will then coalesce into a specification for an overhaul of the platform-native i18n functionality.

The Problem(s)

Types of translations in DHIS2

There are three primary categories of strings in

1. Server-bundled translations For a long time, all translations were bundled into the DHIS2 core and accessible through an API endpoint. Pretty much all use of this system has been eliminated, but some strings still have their translations baked into the WAR file (notably bundled application names)

2. Application-bundled translations ...TODO

3. Dynamic translations Some strings are truly dynamic, created by the implementer of the instance when configuring their metadata model. These strings are created, updated, and accessed through the API and stored in the database. We won't be directly addressing this use-case in for application-bundled strings, but it's good to keep in mind.

Also: Android-bundled translations
Also: legacy i18n system in scheduler app (see PR discussion) which only supports English and Urdu

A brief history of application-bundled translations

1. d2 (the ancient solution) -

2. d2-i18n (the legacy solution) -
   Some info here: https://docs.google.com/document/d/19EK7XoMdzu4E0MMWOZMz1z3G55DJ2dmcMp5HUcggRfo

3. d2-app-scripts i18n (the current solution) -

There are a few straggling uses of the (relatively) ancient d2 system in some of our libraries (notably d2-ui). I won't address the challenges with that system, as it is deprecated and should be removed.

There may be some additional methods of translation out there, such as the one in scheduler-app which only supports English and Urdu and doesn't appear connected to Transifex (see dhis2/scheduler-app#46)

Challenges with the d2-i18n "legacy" solution

Until very recently, the "standard" fleet of internationalization tools for DHIS2 libraries and applications was :

• d2-i18n
• d2-i18n-extract
• d2-i18n-generate

...TODO

Challenges with the "current" d2-app-scripts i18n system

This is almost a direct port of the previous system. It combines ported versions of d2-i18n-extract and d2-i18n-generate into a single App Platform cli module (d2-app-scripts i18n) and automatically runs the extract and generate steps when building an application or library with the platform. The components are:

• d2-i18n
• d2-app-scripts i18n extract
• d2-app-scripts i18n generate

And here are some of the challenges:

1. namespacing - we currently require all strings in all libraries to be under the same namespace (default). This is because the string requests don't include their own namespace (i18n.t('This is a test') not i18n.t('namespace:This is a test')). @jenniferarnesen wrote up a nice overview of how we might re-introduce namespacing here. This would require changing all instances of i18n.t in our sourcecode to include a namespace, and would require us to make sure the chosen namespace separator could never occur in the English string (: is particularly problematic for this reason)
2. sentence keys - we currently use the full English string as the i18n translation key in .pot and .po files, and when extracting strings from sourcecode in the extract step. This can be problematic if that english string happens to include the configured namespace separator or plural separator (: and _, by default) which can happen. Instead, we should probably create a non-crypographic hash (sha1, for instance) of the English string and use that as the key - Transifex should pick up the default English translation and offer that to translators, so it should be an opaque change.
3. plurals - we haven't traditionally supported plural translations, which is problematic. @jenniferarnesen implemented an initial solution to this, but we should ensure it is a first-class feature of the overhauled i18n solution.
4. extraction thrash - we often run the i18n extract command on pre-build and pre-start in applications, but this can lead to a couple subtle issues. Firstly, if the developer doesn't re-run build or start before committing code then the committed pot file will be "out of sync" with the strings in the source code (this can be alleviated with a pre-commit hook i18n extract && git add ./i18n). In some cases, string extraction will also unnecessarily re-order keys in the .pot file and/or will cause only the timestamp in the .pot file to change. This could be avoided by removing the timestamp and ordering keys in a stable way.
5. generated code - there are several issues related to the fact that the i18n generate step generates javascript code in the application (or library's) /src/locales directory. Not only is this confusing and error prone (invalid javascript can sometimes be generated when locales include a -, for instance) but it requires the library or app developer to manually include the ./locales/index.js at least once for each entrypoint, and to mark it as code with side-effects.
6. static module initialization - today, i18n strings need to be initialized AFTER the user's locale is loaded. This is often not done in practice since we sometimes do const str = i18n.t('Hello world') at module initialization time, which means it always uses the English string.
7. right-to-left text - it's currently unclear how we support interpolation in right-to-left languages, which leads to confusion. This is also the reason we initially avoided using react-i18next for VDOM-integrated translations.
8. styled sub-strings - it isn't possible to, say, apply a bold style to a sub-string when using i18n.t. react-i18next supports this, but was initially avoided because of challenges with right-to-left support

Side note : .pot files are likely a relic from an older i18n system, and it might be possible to skip them as an intermediary in favor of pure json files

[SPEC] Requirements for a revamped i18n solution

...TODO

Lazy-loaded locale bundles or SSR

• Today, the locale bundles (all translated strings) are included in the main application bundle. This can be quite expensive when there are lots of strings and lots of translations. The source files for all translations of the Dashboards app (relatively light on strings), for instance, weigh about 95kb (uncompressed)
• Similarly, localized moment bundles should for locales not in use should be omitted from the bundle (this is already done in the app platform).
• One approach to solve this is to use a dynamic import for locale strings once the user locale has been determined.
• This introduces another problem, however, which is that dependencies of an application might have their own translations which cannot easily be lazy-loaded. To solve this, the application would need to "aggregate" all translation strings from all dependencies into singular locale bundles.
• Another option to explore might be a form of SSR (although it could be done at compile-time) which would enable a "localized" version of the application (with a single locale pre-loaded) to be generated and cached. This presents another challenge, however, as the server will need to know the user's locale at the time the HTTP request is received (which could possibly be determined from the session cookie or auth token, if present)

Dynamic locale selection

• It should be possible to dynamically switch locale without changing the user locale. This could be a simple ?locale=es query-string, which would also allow for automated testing of localization in Cypress, for instance.
• It might also be worthwhile to allow dynamic locale selection in the profile dropdown menu, so that it is accessible from any application. This could either update the user's locale or temporarily redirect to the current page with ?locale=<xyz>

Semantic Markup - ref Declaring language in HTML

• The html lang attribute should be correctly set (<html lang="nl">), this will likely need to be done dynamically after page load unless we implement server-side rendering.
• Page elements which contain text in another language than the one set on <html> should set their own lang tag (this is particularly important for untranslated text and for locale names)

CSS - ref CSS for internationalization

• Different languages require different styling for things like text direction (right-to-left, top-to-bottom) and emphasis. CSS rules targeting these languages (using :lang selectors) should be standard, and non-latin languages should be tested regularly.

A proposed design

...TODO

Things left unaddressed by this spec

• Multi-calendar support and date localization (Moment.js is going into maintenance mode, we should reconsider using it with dayjs, which is largely API-compatible)
• Number, Date and list formatting (currently solution is to use Intl.NumberFormat, Intl.DateTimeFormat and Intl.ListFormat respectively)
  • e.g new Intl.NumberFormat(i18n.language).format(value)
  • Need to wrap in try/catch as DHIS2 locales (passed as first argument) are not always ISO 639 compliant

[ismay]

We discussed some i18n things @Mohammer5 and I encountered here on slack. We were discussing the usecase for namespaces, and @amcgee noted that even though the string might be the same, the context could mean that the translation would be different. But we don't really provide context to translators. So one thing we could add is that we should start adding context for translators.

[amcgee]

Related - static analysis of code to find untranslated UI strings dhis2/cli-style#256 (Thanks @ismay @martinkrulltott @HendrikThePendric)

[amcgee]

See also dhis2/app-platform#98 (closed)

I went halfway down the road of building a babel plugin to inject the namespace into the string key of i18n.t calls, but I stopped because it might not be the right approach - we may instead want to stub out the import ‘@dhis2/d2-i18n’ module and replace it with something we control, something that has a fixedT with a set namespace. Here’s my half-complete plugin, worth further investigation...

https://astexplorer.net/#/gist/a5bd7b8c733fd52e3b51d3f713b5a3d3/6b34483cb68fec86c57bd6c3cf7b32dd4995871e (edited)

[Mohammer5]

Another requirement proposal:

It'd be nice to be able to use markup / react components inside the translation string. i18n-react (or so) has a solution for this already. @cooper-joe made some designs for the new maintenance app where there were some styled withing translatable text (like bold text)

[amcgee]

It'd be nice to be able to use markup / react components inside the translation string. i18n-react (or so) has a solution for this already. @cooper-joe made some designs for the new maintenance app where there were some styled withing translatable text (like bold text)

@Mohammer5 definitely! The challenge with react-i18next was at one point that it didn't have good support for right-to-left text, I believe, but it's worth re-evaluating!

[ismay]

See also this thread: https://dhis2.slack.com/archives/C0BP0RABF/p1601970390067700. We also need to support localised pluralization, and allowing for translation context would be good as well. Currently I don't think we properly support that (but correct me if I'm wrong).

[jenniferarnesen]

Regarding plurals, there is a problem with our current set up that is short-circuiting translations. This is currently happening for the interpretations component in d2-ui.

Locally, the en.pot file is extracted from the code. Since English uses the keys as the translation, and plural keys have a "_plural" appended like this: {{count}} reply_plural, our code needs to actually provide English translations for the singular and plural versions of the string. This can be done in the code like this:

const replies = i18n.t("{{count}} reply", {
        count: repliedBy.length,
        defaultValue: "{{count}} reply",
        defaultValue_plural: "{{count}} replies" 
    });

Once extracted, the en.pot file looks like this:

msgid "{{count}} reply"
msgid_plural "{{count}} reply"
msgstr[0] "{{count}} reply"
msgstr[1] "{{count}} replies"

But transifex does not accept pot files with translations in them (the msgstr[0] and msgstr[1]). So currently, the process stops here and updated po files are not being created on transifex.

One potential workaround is to post-process the en.pot file in the extract step, so that we end up with this:

msgid "{{count}} reply"
msgid_plural "{{count}} replies"
msgstr[0] ""
msgstr[1] ""

Then we can configure transifex to push an en.po file (which it currently doesn't). The en.po file would look like this:

msgid "{{count}} reply"
msgid_plural "{{count}} replies"
msgstr[0] "{{count}} reply"
msgstr[1] "{{count}} replies"

Which would be generated into the en/translations.json file:

"{{count}} reply": "{{count}} reply",
"{{count}} reply_plural": "{{count}} replies",

Phil and I have been discussing this and agree that it would be very strange if we are the first organization to be running into this problem. So is there something I'm missing?

One interesting fact - during the extract-pot process, i18next-scanner actually generates the needed json for en/translations.json (this json is then used to create the pot file):

"{{count}} reply": "{{count}} reply",
"{{count}} reply_plural": "{{count}} replies",

So maybe there are three solutions, given our current configuration:

1. post-process the en.pot file and configure transifex to create an en.po
2. during the build step (maybe pre-build), run i18next-scanner on en.pot and copy the resulting en/translations.json to src/locales
3. send and en.po instead of en.pot file to transifex

[jenniferarnesen]

So, just to follow up, the d2-ui interpretations component now has a modified extract script that applies default values for all the strings. This gets around the transifex issues. Here is the script: https://github.com/dhis2/d2-ui/blob/master/packages/interpretations/scripts/extract.js
@amcgee

[jenniferarnesen]

The solution is to still send an en.pot file to transifex, but now all the default strings are filled out: https://github.com/dhis2/d2-ui/blob/master/packages/interpretations/i18n/en.pot

[paschalidi]

Without having a deep knowledge on the topics as you all do, I wanted to drop here two links.

1. fbt which is basically fb's open source library that handles translation with taking grammar between different languages in account.
2. A short tech talk on the ftb library and its capabilities. https://www.youtube.com/watch?v=7XW3SkWsaKo

Some context on why I am actually adding this. Hoping that it can serve us some sort of inspiration in out own problems. So that perhaps seeing how fb handles their translation we can get into a similar solution ourselves if wanted to build our own in-house solution for this.

Happy to help further with this as I can :)

[ismay]

Whilst working on the sharing dialog I noticed we don't currently have a way to translate a text with multiple pluralizations. So for example: "{{ amountLikes }} likes for {{ amountArticles }} articles". Also, even a text with a single pluralization currently doesn't work out of the box (as Jennifer mentioned).

On a related note, whilst going through this I started wondering if it might be viable to consider other upstream libraries (so besides i18n). i18n seems to be actively maintained, but mainly just by two people. Lately just one even. So documentation is lacking for example. As Christos mentioned, there are other libraries out there (like fbt, etc.), which might be a better fit than i18n for our purposes, and more stable and easier to use over the long run.

[mediremi]

Nice video on fbt by one of the developers: https://www.youtube.com/watch?v=X6uT-wvqfNo