# Implement language fallback #492

Closed
opened this issue Jun 6, 2013 · 14 comments

Projects
None yet
5 participants

### mattflaschen commented Jun 6, 2013

 For instance, if the code requests a locale of de-CH, it should fallback to de, since currently there is no de-CH. This is an IETF language tag, an extension of ISO 639-1. That way, you can just request any language without knowing what's implemented. And if a de-CH localization is added later, it will just work a little better, without the client code changing anything. Fallback to English already works for when you request something like zh which isn't implemented at all. You may be able to import the fallback configuration from MediaWiki. They have fallbacks configured (e.g. https://git.wikimedia.org/blob/mediawiki%2Fcore.git/d7a61d72231861755f0ccdaf80d7b4388cebd5df/languages%2Fmessages%2FMessagesMwl.php has a fallback to 'pt'), except when the fallback is to English. Note that they are often circular, though ('pt' falls back to 'pt-br', which falls back to 'pt').

Closed

Contributor

### fred-wang commented Jun 7, 2013

 Thanks for the report. Just a digression FYI on the English fallback. The English language files are actually not used, Davide just created these files to make translation possible until we moved to a system like TranslateWiki. When a localizable string is used in MathJax, we use both the string id and the English version. This was to ensure that the English fallback is always available (for example, even when downloading the translation data fails or is not finished) and also because Davide wanted to see the explicit English string in the source code. So by design, the English fallback can always happen when something gets wrong and in particular when the language is not available. Also, this should really be en-US since we are using words like "color" rather than "colour". Regarding the implementation of a general fallback, I remember that there are a HUGE number of languages when I studied the HTML5 validator. So I'm wondering if (arguably naive) fallback implementation like the following could help: when one specify "xx-yy-zz" try it first, next try "xx-yy", next try "xx" and finally the English fallback? I realize that the case of "en-US" and "pt-bz" that exist before "en" and "pt" show that it won't work in general. So perhaps this can be combined with another non-automated method: just specify some mappings in the MathJax.Localization structure that will tell e.g. to use "pt-bz" when "pt" is asked and could be configured as wanted (perhaps via a function setLanguageFallback or something). That way, we won't need to add language data or redirection for all the existing languages, only those that are needed.

Contributor

### fred-wang commented Jun 17, 2013

 So I've just added a "remap" entry in the default language data in order to specify an existing fallback language. For example  pt: {menuTitle: "portugus\u00EA", remap: "pt-br"}, "pt-br": {menuTitle: "portugu\u00EAs do Brasil"}  will tell MathJax to use "pt-br" as a fallback for "pt". The remap is computed by the getLocale function. If necessary, this could do something more clever as the heuristic I mentioned above. But I think it's ok for now.
Member

### dpvc commented Aug 1, 2013

 I think your idea of an automated fallback algorithm makes a lot of sense, especially when combined with the remapping that you already have (which can be used to move from less specific to more specific, as with pt -> pt-br). I would also suggest allowing something like pt: {remap: "pt-br"}  so that you can remap a locale without having it specifically in the MathJax menu. (The createLocaleMenu method in MathMenu.js would need to be modified to account for that.) Not that I'm saying to do that for "pt", but just to allow a generic remapping facility that isn't directly tied to the menu. So I would say go ahead and implement your scheme. That will allow   to use the "en" locale (the default), or locale=pt-XX for other XX to first go to pt, which remaps to pt-br.
Member

### dpvc commented Aug 1, 2013

 Your issue495 branch adds a getLocale function, but I think you can fold that into the setLocale method. Currently, the locale must be looked up every time a string is needed (e.g., for every "Processing ... " message), but it really only needs to be done once, when the locale is first determined. The MathJax.Localization.locale is not the one that controls the menu selection, so it is OK to set this to the actual locale used, even if it is remapped. The menu will still show the one selected by the user. It will be more important once you do the algorithmic lookup, as that would make getLocale more expensive.
Contributor

### fred-wang commented Aug 2, 2013

 I've created a new branch issue492. => Ready for review.
Contributor

### fred-wang commented Aug 7, 2013

 => UnitTest wanted
Member

### dpvc commented Aug 7, 2013

 I did a little more testing, and there are two lines in MathJax.js that need to be changed to call MathJax.Localization.setLocale() rather than setting MathJax.Localization.locale directly. Line 2300 needs to be MathJax.Localization.setLocale(this.params.locale);  and line 2379 needs to be {MathJax.Localization.setLocale(config.menuSettings.locale)}  The one difference is that if MathMenu.js is already loaded, this will queue the loading of the locale's MathMenu domain file. I don't think this will be a problem, but it is something to keep in mind for future bug reports. Alternatively, you could make a copy of setLocale and rename it something like resetLocale then remove the last line about loading the MathMenu domain. Then make setLocale call resetLocale and then do the loading of the domain. Finally, use resetLocale for the two lines above. (Whew!)
Contributor

### fred-wang commented Aug 8, 2013

 I added tests for setLocale and &locale= API/Localization/param-1.html API/Localization/setLocale-1.html => In testsuite

### dpvc pushed a commit that referenced this issue Aug 8, 2013

 Merge pull request #539 from fred-wang/issue492 
Resolves issue #492
 99d7fed 
Member

### dpvc commented Aug 8, 2013

 ==> Merged

### dpvc pushed a commit that referenced this issue Aug 8, 2013

 Merge pull request #540 from fred-wang/issue529 
Resolves issue #492
 c5f8c34 

Merged

Contributor

### fred-wang commented Oct 8, 2013

 Currently, the fallback is applied when the whole translation for a language is missing. I think that now that "pt" is partly translated, it won't fallback to "pt-bz" but uses "pt" for strings that are translated and "en" for the untranslated strings.

### he7d3r commented Oct 8, 2013

 That is not good :-(
Member

### pkra commented Oct 8, 2013

 @helder-git do you have another suggestion?
Member

### dpvc commented Oct 8, 2013

 @fred-wang, could we copy any missing entries in the pt file from the pt-brfile when the pt files are built, so that they contain no untranslated strings? That is, do the defaulting to pt-br up front when we create the pt files from the TranslateWiki data? Alternatively, we could not include pt until all the strings are available (so that MathJax selects pt-br when pt is requested).

### he7d3r commented Oct 8, 2013

 Isn't possible to have something like englishDefault = { msgKey1: msgText1InEnglish, ... } fallback1 = { msgKey1: msgText1InFallback1, ... } //... fallbackN = { msgKey1: msgText1InFallbackN, ... } userLanguage = { msgKey1: msgText1InUserLanguage, ... } and then make something like \$.extend( englishDefault, fallbackN, ..., fallback1, userLanguage ) ? If not, @dpvc's first suggestion seems good for now.

Merged