Kibana Globalization

[srl295]

Kibana Globalization

Note: These requirements were originally written up over a year ago. Take any specific implementation details with a grain of salt. Phases that were already implemented should be reevaluated to make sure they still make sense, especially in light of the new kibana platform work.

Purpose

• Provide translation readiness in Kibana for view text (user data is a separate issue)
• A core translation layer that is framework agnostic and consumable in all parts of Kibana
• Provide plug-and-play capability (i.e. code free) for localization engineers when providing translations of different languages
• Delivered in a phased approach

Architecture Diagram

This was our original plan, but it might need to change dramatically for the new platform

Design

Phase 1

Note this phase was completed, but it's possible the implementation might need to change for the new platform. Most of the code lives here

Purpose

• Implement the internationalization engine (I18n class) which provides a means to supply translations in a standard format that it not dependent on a localization framework
• i18n engine manages all locale translations; registering all translations and loading the correct locale when required
• Translate the Kibana welcome message which proves that the I18n class provides translations as registered for Kibana

Decide Language for Translation Algorithm

• The algorithm which decides the language for loading translations is as follows:
  1. First, do a direct comparison with the highest priority locale in the HTTP header “accept-language” against the registered translation languages. If comparison found then this is the language returned.
  2. Next, do a partial comparison of this locale whereby you try and get a language code comparison against the registered translation languages. For example, "fr" or "fr-FR" is used for all regions of French if comparison applies. If comparison found then this is the language returned.
  3. Next, repeat step 2 and if need be step 3 for the next highest priority locale in the “accept-language” list.
  4. Continue, until match found in step 2/step 3 or end of the “accept-language” list. If no match found then return empty language.

I18n Class

• Manages the language translations for Kibana

• Responsible for loading translated content per language

• The translations file are JSON files with a flat keyspace, where the keys are unique. This uniqueness between translation plugins could be achieved by prefixing the keys with the plugin name. The key signifies the translation ID which would be referenced in translatable files (like JS, HTML etc.).

• The key value is the translation string

• Example translation JSON file

en.json

    {
        "UI-WELCOME_MESSAGE": "Loading Kibana",
        "UI-WELCOME_ERROR": "Kibana did not load properly. Check the server output for more information."
    }

• Core Kibana plugins like ‘kibana’ and ‘status_page’ could come with their own English translations bundled in
• API:
  • Return all translations for registered locales
    • getAllTranslations()
    • A Promise object where keys are the locale and values are Objects of translation keys and translations
  • Return all translations registered for the default locale:
    • getTranslationsForDefaultLocale()
    • A Promise for an object where keys are translation keys and values are translations
  • Return translations for a suitable locale from a user side locale list of BCP 47 language tags:
    • getTranslations(...languageTags)
    • A Promise for an object where keys are translation keys and values are translations
    • This object will contain all registered translations for the highest priority locale which is registered with the i18n module
    • This object can be empty if no locale in the language tags can be matched against the registered locales
  • Register translations:
    • registerTranslations(<absolute_path_to_translation_file>)
    • The path to the translation file is registered with i18n class

UiI18n Class

• Handles the interaction between the UI/server and localization (I18n class)
• Maps the accept-language header to BCP 47 tags
• Fetches the language as requested where the locale is decided by the translation algorithm
• It also substitutes any missing translations with the default locale translation

Tool for Verifying All Translations have Translatable Strings

• Grunt run task tool that tests all translatable strings have a translation (i.e. all translation IDs have a corresponding translation string)
• This could be run by CI to verify globalization end-to-end capability
• A possible solution:
  • For non-angular constructs: Enforce a pattern to be used. For example a i18n(<key>) function in the Jade template. A tool can then be used to find such pattern and extract the keys to file
  • The keys in the key file(s) would then be checked against the language translation files registered

Deliverable

• Translate the start-up message (“Kibana is loading ...”) and error message in the Jade template
  (Kibana Jade template)
• Translation message will be loaded using the language as per the deciding algorithm. For phase 1 the default 'en' langauge will be used.
• I18n unit tests
• Tool can be run by CI to verify translation IDs have a corresponding translation string
• Kibana core plugin registers its translation file during the initialization phase. The translation file contains strings for the welcome message and the start-up error message.

Phase 2

Purpose

• Integrate a framework like i18next or formatJS with the i18n engine, or build our own. i18n framework needs to be independent from any UI framework (angular, react, etc).
• Provide example for enabling translations in an AngularJS view and a React view by creating a POC for each
• Implement a tool to generate the translation plugin
• Documentation

Translations Available Client Side

• Make the translations available on the client side by:
  • Embedding the translations in the initial HTML payload:
    • Reduces complexity and removes a round trip
    • Evaluation required to test if it performs well and does not create a large payload
  • If the evaluation of HTML payload is non performant, then the following approaches should be investigated in this order:
    1. Use current bundle mechanism:
       • (Kibana Jade template) to call API and generate the bundle which will be loaded during start-up
       • The JavaScript bundle produced will be of the following format: i18n_<language>.bundle.js
       • Kibana loads all resource bundles on the client side after starting the single-page application
    2. Client side directly calls REST API and loads the JSON payload:
       • GET /i18n/translations ==> returns the English (or German or whatever) translations negotiated with the browser HTTP header “accept-language” priority list compared against the languages supported
       • GET /i18n/translationsForDefaultLocale ==> returns the default locale translations
  • All approaches will decide the language to be served up by using the language deciding algorithm (from Phase 1)

Translation Identifiers Added to Kibana UI

• Ids are added to the relevant view with Angular and HTML content
• English (en) translation file is generated for the Ids defined

BEFORE (HTML)

    <div class="sidebar-list">
    …

    <div class="sidebar-list-header">
        <h5>Selected Fields</h5>
    </div>

AFTER (HTML)

    <div class="sidebar-list">
    …

    <div class="sidebar-list-header">
        <h5 translate="FIELDS_SELECTEDFIELDS"></h5>
    </div>

Tool for Verifying All Translations have Translatable Strings (Update)

• Verify translation IDs in Angular templates
• The keys in the key file(s) would then be checked against the language translation files registered

Translation Plugin Generator

• Tool which generates a translation plugin
• Localization engineer should only need to add translation file(s) within the plugin directory, refernce the file(s) in the 'uiExports' plugin extension, and add plugin to Kibana
• All integration with Kibana works OOTB

Deliverable

• Angular and React views where globalization is enabled showing the pattern to use
• Verification tool updated to check translation keys in UI
• A generator that localization engineers can use to produce language translations and integrate them in Kibana in an easy manner
• Documentation relevant for the following stakeholders:
  • those that want to enable translation in Kibana
  • those that want to contribute translations to Kibana
  • those that want to create a Kibana Plugin

Phase 3

Purpose

• Add the translation identifiers and English translation strings for the existing Kibana views

Translation Identifiers Added to Kibana UI

• Ids are added to all the relevant UI content (HTML, JS etc.)
• English (en) translation file(s) are generated for the Ids defined

Deliverables

• English translation bundles are available so that localization engineers can start generating translation plugins for different languages

Phase 4

Purpose

• Contributing language packs

Handling Language Packs

• Provide a pseudo language which will test the potential real estate of views for all language
• Define core supported languages
• Process for language pack builders:
  • Creating packs
  • Merging packs

Deliverables

• Process for contributing language packs
• Core language packs as supported by Kibana

Blockers

Open Questions/Issues

• Translation of user data. TODO: requires a separate issue.
• Server side set locale (ignore the client's requested locale)
• Provide mechanism in UI for user to manually switch their language from list of supported languages
• What about bi-directional language support? This affects language, charts and UI content support.
• UI views should be able to handle different languages (This means when switching from one language to another (for example, English to German) that the look and feel is maintained) :
  • Different languages can have variable content lengths. This means having adequate spacing to handle the strings in each language.
  • Need to consider different fonts. Some web fonts won't support all possible languages.

History

This section has links to prior versions of this issue text.

• 2018 March 7th
• 2016 December 20
• 2016 December 14
• 2016 August 3
• 2016 August 2
• 2016 July 26
• 2016 July 20
• 2016 March 11

Authors

• Matt Bargar
• Martin Hickey: @hickeyma
• Scott Russell: @DTownSMR
• Shikha Srivastava: @shikhasriva
• Steven R. Loomis: @srl295

[rashidkpc]

    <div class="sidebar-list-header">
        <h5>{{ FIELDS_SELECTEDFIELDS | translate }}</h5>
    </div>

Is this expecting FIELDS_SELECTEDFIELDS to be attached to $scope of the controller? It would make more sense to implement this as an attribute directive similar to ng-bind

eg:

     <div class="sidebar-list-header">
         <h5 kbn-bind="FIELDS_SELECTEDFIELDS"></h5>
     </div>

Then kbn-bind would know how to handle FIELDS_SELECTEDFIELDS as a string, retrieve the appropriate replacement text, and fill it. In addition it would allow us to use one time binding and not suffer the performance consequences of two way binding with a filter.

[rashidkpc]

    {
        "FIELDS_SELECTEDFIELDS": "Selected Fields",
        "METRIC_COUNT": "Count"
    }

My initial thought is that a massive master list doesn't seem very elegant. It seems like this sort of thing would be better implemented the way we do .less files, scoped to the current app?

[rashidkpc]

There would also need to be a test framework around this to ensure that all of the constants that must be defined, are. Potential translation volunteers need a simple way of discovering what needs translation and a simple way of confirming that they've defined everything required

[rashidkpc]

    $translate('METRIC_COUNT').then( function(count) {
        return new MetricAggType({
    title: count
    }}

This seems to imply that any functions that currently contain strings would need to be updated to return a promise. I don't see the need for the async behavior here.

[rashidkpc]

• Use angular-translate

Given the need to translate both angular and non-angular components, why was a framework specific translation implementation chosen? And why this one specifically?

[srl295]

@rashidkpc thanks. Comments on a couple of items:

• translation scoping - yes, the translations could definitely be scoped to smaller units. I'll take a look at how the .less files are scoped.
• async behavior / angular-translate - this was one option. There may be a simpler option. The non-angular components might be able to use a different mechanism for loading content. We picked angular-translate to try and use something pre existing for the angular side.

[DTownSMR]

Rashid, maybe I should take these out of order.

• Why a framework & angular-translate? - The angularjs side of things looked to us like a
  good place to start a first iteration of efforts of adding g11n to Kibana. As Steven
  mentioned we picked angular-translate since we didn't want to reinvent the wheel and it
  is feature rich, good community support and seems to be gaining momentum within the
  angularjs community. We're not married to that solution of if there is a different
  preference within the Kibana community we can certainly look at that.
• angular-translate supports a "translate" directive in addition to it's filter mechanism.
  We were just showing an example of what translation can look like.
• angular-translate does not require a single resource bundle. It supports a number of
  options. Their partialLoader that will handle breaking up the resources at a component
  level.
• The idea of a test framework is interesting, especially for on-going efforts.

[shikhasriva]

Rashid, regarding test framework, we suggest having that as a separate issue. And definitely agree with the need and usage of it... what do you think ?

[shikhasriva]

Rashid and all, DO you all agree with the phased approach we suggested ...thanks

[w33ble]

Given the need to translate both angular and non-angular components, why was a framework specific translation implementation chosen?

While I agree that angular-translate looks pleasant to use, and seems to be well maintained, using something that's explicitly bound to angular for something as far-reaching as i18n concerns me.

As @rashidkpc points out, we have non-angular components in our code, and that's likely to become increasingly common in the future. I'm also concerned about server-side use as we push things into the node server.

Is there something more generic that fits the bill, and also has an existing angular wrapper?

[rashidkpc]

I haven't looked deeply into i18n, but I'm hesitant to tie to angular-translate for the full implementation. It could be used in places, but there's plenty of places it can't be used. Like joe said, I think it would be good to investigate something lower level that could be wrapped up for the angular bits.

On the topic of the phases. I don't see any problem with those stages, but i think more discussion is needed on the approach before beginning.

I would like discussion of the testing framework to happen here. We can always split it off if needed, but it has direct influence on the approach taken.

[srl295]

The translated files are just JSON and could certainly be consumed by the server side or the widgets, etc without use of the angular-specific framework. So this approach doesn't preclude using angular-translate where it makes sense, and something else (or no framework) elsewhere.

Re testing - yes, we can definitely discuss it here. We were wanting to split the test itself into a separate phase as above. (I added 1AA) One issue would be that I would recommend having a missing translation be more of a warning than a failure from a CI perspective, otherwise a feature couldn't be added in the source language (English) without all translations being done, nor could a translation be partial.

[w33ble]

The translated files are just JSON and could certainly be consumed by the server side or the widgets, etc without use of the angular-specific framework. So this approach doesn't preclude using angular-translate where it makes sense, and something else (or no framework) elsewhere.

That's a fair point. It does seem like the format is pretty common, and the more or less the same regardless is which library/framework you use. Getting hung up on angular-translate specifically is probably a waste of time.

Given that, I think it's worth adding something in one of the earlier phases about ensuring that whatever translation format we go with works both with and without Angular somehow, most likely by specifically targeting a part of the application that isn't in Angular. Server-side code is probably an easy candidate.

[srl295]

@w33ble OK. Perhaps something like - "a simple JSON format will be specified in the documentation, and the proof of concept will use angular-translate. Non-angular portions (such as the server) would not use angular-translate, and the specific plugin technology used for translating on the client side is subject to discussion"

[w33ble]

I think Rashid and I just want to make sure the plan specifically takes into account using the i18n definitions outside of Angular. Adding the above sounds good to me.

[srl295]

@w33ble — thanks, that makes sense.
Edit I have updated the text above to address the feedback from yourself and @rashidkpc .

[srl295]

@rashidkpc @w33ble — any comments on the updates?

[rashidkpc]

I'm happier with the new proposition.

    var uiStrings = …; // loading TBD
    return new MetricAggType({
      title: uiStrings.METRIC_COUNT,

Either webpack/import or though a Private module, which would make it easy to inject into other parts of the application.

[srl295]

great. We will investigate using webpack.

It seems like the proof of concept should be able to proceed for phase 1A if there are no issues on that part.

[Bargs]

Can we create a new ticket specifically for Phase 1A and turn this ticket into more of a meta ticket? There's a lot of good discussion and context here so I wouldn't want it to get closed when #7247 gets merged.

[hickeyma]

Great stuff @Bargs and @spalger. Lets try and sync-up on IRC and work out how to progress for phase 2 and 3.

[hickeyma]

@Bargs, @epixa, @shikhasriva, @spalger, @posijon Thank you all for attending the sync-up yesterday and for your feedback.

Here are notes from the meeting (let me know if I have misrepresented anything):

• It is good that we landed Phase 1. We want to now drive on with Phase 2 and 3.
• Phase 2 is to get the angular template right for localization engineers to use.
• Phase 3 is to add translation bundles for various languages.
• Updates to Phase 2:
  • In "Deliverable" section, "making the translations to client side" needs to be updated. The proposal is to now implement it by embedding the translations in the initial HTML payload. If discovered that this implementation is non performant then we will consider other approaches like: using current bundle mechanism or then using REST APIs to return the translations.
  • Add to "Deliverable" section, Elastic needs to handle how to translate and verify xplugins with React
• "Tool for verification of translations" needs to be moved from Phase 3 to Phase 2.
• "Implement a tool to generate the translation plugin" is to be moved from Phase 3 to Phase 4.

[hickeyma]

@Bargs, @spalger Updated the design doc after yesterday's meeting as kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[Bargs]

Updated

[yuwtennis]

+1

[hickeyma]

@spalger Updated the design doc kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[spalger]

Could you update the design doc when you get a chance pls.?

Sorry, which design doc are you referring to?

[srl295]

@spalger he's asking for this issue ^^ ’s text to be updated from the above gist. As issue creator I was able to do it - so Done

[hickeyma]

Thanks @srl295

[hickeyma]

@spalger unfortunately no rights on the doc to update!

[srl295]

FYI: 🎦 posted blog+movie showing off g11n progress so far https://srl295.github.io/2017/03/17/translating-kibana/

[tangyong]

Great work so far! @srl295

[elkusr]

Thanks for all the great work! When can we expect phase 3 to be available?

[epixa]

@elkusr There's no target yet for phase 3, and there's still some work in phase 2 to complete first now that Kibana is using react so much.

[zzvara]

How are these efforts progressing now?

Is that correct, that the tasks presented in (https://srl295.github.io/2017/03/17/translating-kibana/) can be used to add any language to a Kibana instance (for example 5.6)? In other words, adding another language to Kibana is feasible as of this writing, it just takes a "little" effort, and that Phase 3 is about to ease this effort?

[epixa]

@zzvara It isn't possible to translate Kibana right now. Phases 2 and 3 need to be completed before you can translate templates across Kibana.

[sunneyoy]

Looking forward to the Chinese version.

[udyr-f]

Have you already supported Chinese?
config/kibana.yml
i18n.defaultLocale: "zh_CN"
This configuration does not work, so I use this tool now.
Kibana_Hanization

[kimjoar]

@sunmaolin Sorry, we currently only support English.

[epixa]

I'm going to close this issue out in favor of #17201. While that is a much newer issue than this one, it contains a lot of information about our current localization effort, and it's best if we don't fragment the discussion across multiple issues.

Many of the details in that other ticket are based on all the things we learned from this effort in particular, so thank you to the great folks at IBM and everyone else that has been involved in getting us to this point!