Translation Enablement - Phase 1A

[srl295]

• documented in TRANSLATION.md
• using angular-translate
• English strings in src/plugins/kibana/public/settings/sections/indices/i18n/en.json

Fixes: #7255

Author: Scott Russell scott_russell@us.ibm.com @DTownSMR
Author: Martin Hickey martin.hickey@ie.ibm.com @hickeyma
Author: Steven R. Loomis srloomis@us.ibm.com @srl295

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[srl295]

/cc and thanks @shikhasriva

[srl295]

• At this point, there isn't fallback language support. So if your browser requests es and it's not installed, you get raw strings.

[srl295]

• failures — not quite sure what they mean. Can someone help explain?

[DTownSMR]

Steven, wasn't Martin's change included that supported fallback to English?

[srl295]

@DTownSMR yes - but it required a hard coded list of locales. Looking at how to avoid that.

[Bargs]

The test that's failing is one of our browser based unit tests. You can run this locally with npm run test:dev which will launch a karma instance and a browser that you can use devtools in to debug the test. Haven't had a chance to look at it closely myself but I didn't see an obvious connection between the failing test and the changes here, so I'd be interested to know if the same test fails for you locally.

[hickeyma]

@Bargs: Thanks for the tip. I ran the unit tests as you suggested and one of the tests is failing. It seems to be "debounce service -> cancel". I have added the error output as follows: http://dpaste.com/3PHSWRC

[srl295]

Debounce failures fixed in 5d9959c (see #5446 )

[Bargs]

jenkins, test it

[srl295]

^ 2337/2337 passed, but the server exitted with failure.

[rashidkpc]

jenkins, test it

[srl295]

@rashidkpc @Bargs thanks- what's next?

[Bargs]

@srl295 is the code ready for review?

[srl295]

@Bargs I think the core code and approach is ready. But actually, I think the fallback support (noted above) should be in. @hickeyma is working on some tests as well.

• tests
• replace 'TODO' in docs

[Bargs]

This should instruct the user to fork the Kibana repo rather than just checking it out.

[srl295]

Good point.

• fork, not checkout.

[srl295]

docs updated

[Bargs]

@srl295 I started looking at the code a bit today, I'll try to dig in more on Tuesday next week and provide some more feedback.

[epixa]

I haven't given this a thorough review, but I do have one immediate concern: by translating the template strings via directives/filters during runtime rather than by a consistent process at build time, we're limiting translation support to only angular templates, and we make it much harder to develop tooling around the translation system.

We don't have a plan to move away from angular at the moment, but we definitely don't plan to be on angular 1.x forever. As for tooling, I think a bare minimum for a translation system would be the ability to programmatically find missing translations in a language file, which Kibana core CI builds would outright fail on if there were any missing english translations. I'm not sure how reliable that would be when translations are applied at runtime.

[srl295]

@epixa Thanks for the feedback. Please see in #6515, we aren’t tying this solution to Angular in any way. The files are just JSON files, and we show examples of non-Angular components using them. We don't have any of these in the PR for this particular phase.

As for tooling, I think a bare minimum for a translation system would be the ability to programmatically find missing translations in a language file, which Kibana core CI builds would outright fail on if there were any missing english translations. I'm not sure how reliable that would be when translations are applied at runtime.

@hickeyma is working on exactly that- automated tooling to compare the translation file (e.g. de.json) to make sure it includes everything that is in the source file (e.g. en.json).

[epixa]

Glad to hear about not being coupled to angular! I should have reviewed the contents of the PR more thoroughly, but I really wanted to get you some initial feedback before I headed off yesterday.

My concern with tooling wasn't so much about ensuring that other languages had the same translations as en.json so much as identifying the scenario where a word identifier is used somewhere in a template and there is no corresponding translation in en.json. This could happen due to a variety of different mistakes, including a simple typo. For example:

// en.json
{
  "GREETING": "Hello"
}

// someview.html
<span translate>GREETIGN</span>

It's imperative that that scenario cause an error at build time so that it wouldn't even pass CI.

[srl295]

Ok. Maybe use https://www.npmjs.com/package/angular-translate-extract for testing.

[hickeyma]

@epixa Thanks for the feedback. I am wondering if testing of the identifiers in the code (JS, HTML etc.) comes under the current testing of Kibana? My thinking is that adding globalization support does not alter the functionality of the product, it added the identifiers and string substitutions for each language supported. If an identifier is therefore incorrectly added, should this not be picked up by current testing as the string will not be returned?

[epixa]

From a practical perspective, I don't think relying on the existing tests is going to be feasible. Getting complete test coverage on Kibana is admirable, but there are practical and technical obstacles to overcome to make that happen, so it's a long-term goal at this point. It could be years before that happens, and I'd love to get i18n support before then. Our approach for testing at this point is to continue increasing test coverage on existing code where possible and to ensure that any new functionality being added is done so in a way that is itself testable, and that would apply to something like translations as well.

Stray identifiers in the UI might not technically impact functionality in the UI, but they have a negative impact on users in a much different, but no less important way. The ambiguity in a stray identifier could make a user uncomfortable using an entire feature, or worse it could make users misunderstand what the feature is suppose to do. We're talking about the possibility of adding delete data functionality in the upcoming pipeline work - for stuff like this, it is absolutely imperative that a missing identifier doesn't make that feature any less clear.

But beyond all of that, I think it's important that we can test the translation system itself without running functional tests on Kibana.

[srl295]

@epixa @Bargs should testing be in this PR or in a subsequent one?

[srl295]

OK Kibana people… I thought I had an idea for how this could work, but now I'm not sure.
Problem: Need to figure out which languages are installed, at some start-up time (build, optimize, serve…) to know what is available and when to fall back to the source language (English).
As this PR stands, if you don't have one of the supported languages chosen (i.e. your browser doesn't request English explicitly), you won't get the correct content.
Example: if you have English and German installed, the list will be [ 'en', 'de' ]. If you just have the English content, it will be [ 'en' ] ,etc. Probably this output would be a small .js file just exporting the array:

module.exports = ['en'];

Options:

1. Have users manually edit an installedLanguages.json file with a static list. (Not recommended, but wanted to list here for completeness.)
2. Have users run some task (npm run update-languages) whenever they add/remove .json files to detect the list of languages available.
3. Run code inside of the src/optimize phase to scan for i18n/*.json files at optimize time. This might be good, because then at optimize time we can also build optimal language bundles ( planned as a later enhancement- see Kibana Globalization #6515 under Phase 1AB )
4. Run something under the src/cli or src/plugins mechanisms that would run early and do the scan/update.

Any thoughts about the preferred approach?

[Bargs]

Testing should definitely be in this PR if it's going to go into master. Really I'd like to see all of 1A(X) (as well as example usage inside and outside of angular) working since building, testing, and in app usage all impact each other. It seems like we're moving in a good direction, but this feature touches so many different things its hard to predict all of the ramifications without seeing at least a POC with all the moving pieces.

I don't have a good solution for creating the supported languages list yet. Optimize seems like a good place, except that the tests will also need to know which languages to test, and ideally they should run without needing the optimizer. A related question: the main GH issue says "Kibana users could "plug in" translation (language) bundles". How will this work?

[srl295]

all of 1A(X)
Thanks, that's helpful.

[tests] should run without needing the optimizer
The tests can run the same file-discovery algorithm to locate the files.

Kibana users could "plug in" translation (language) bundles. How will this work?

As this PR stands now, to add German they can just add the appropriate files. That's why the discovery phase mentioned here is needed.

src/plugins/kibana/public/settings/sections/indices/i18n/de.json
…

[epixa]

I'd like to see the supported languages feature implemented through the plugin system. This would mean that installing a new language means installing a new plugin. That would give us all the tracking capabilities we need for what languages are available, it would allow us to make decisions about languages during Kibana start up, etc.

bin/kibana-plugin install i18n-de

[srl295]

@epixa should (could) the built in English content also be a plugin? Is there any documentation on plugins besides the "PLEASE DON'T WRITE CUSTOM PLUGINS" message?

[hickeyma]

@epixa Thanks for feedback on testing of stray identifiers. It makes sense that the user usability should not be broken. I agree.

Potential test case could be: Loading the list of translation identifiers per localization directory and comparing with the code files (HTML and JS) for that directory for the existence of the identifiers. If identifier is not present then we have a stale or stray identifier.

[Bargs]

@srl295 @hickeyma I realize our feedback is kinda all over the place right now, I apologize for that. Would you be up for a video chat with @epixa and I? A lot of our feedback is driven by our thoughts on the future of Kibana's architecture, so it would be easier to hash some ideas out on a call rather than back and forth on a PR thread. We think this feature is really important so we'd like to give you guys the support you need to implement something really rock solid that'll work well into the future.

I'll try to highlight a few of the high level things we'd like to discuss. Note that none of these future plans are totally concrete at the moment, which is why it's easier to talk about it rather than write about it.

• Over time we'd like to slim down the Kibana core. Even things like Dashboard and Discover will ideally be implemented as isolated plugins. So the most important first step in implementing globalization is probably creating a solid, framework agnostic service that all Kibana plugins can consume and contribute to.
• Angular should probably consume this service rather than the i18n files directly.
• Testing and translation checks should work without needing to run/optimize Kibana. We'd like to get rid of the optimization step in production anyways.
• All translation files should probably be added to the system in the same way, even the default english ones.

[epixa]

@srl295 Is this PR still relevant given the other work you've been doing on the i18n support?

[elasticmachine]

Can one of the admins verify this patch?

[srl295]

Superseded by #7545 (which has been merged)

[liangrui1988]

How do you translate it into Chinese? Thank you

[epixa]

@liangrui1988 Kibana isn't translatable yet. This issue represented an internal initial effort to support a future translation feature.