Implement language fallback

[mattflaschen]

For instance, if the code requests a locale of de-CH, it should fallback to de, since currently there is no de-CH. This is an IETF language tag, an extension of ISO 639-1.

That way, you can just request any language without knowing what's implemented. And if a de-CH localization is added later, it will just work a little better, without the client code changing anything.

Fallback to English already works for when you request something like zh which isn't implemented at all.

You may be able to import the fallback configuration from MediaWiki. They have fallbacks configured (e.g. https://git.wikimedia.org/blob/mediawiki%2Fcore.git/d7a61d72231861755f0ccdaf80d7b4388cebd5df/languages%2Fmessages%2FMessagesMwl.php has a fallback to 'pt'), except when the fallback is to English.

Note that they are often circular, though ('pt' falls back to 'pt-br', which falls back to 'pt').

[fred-wang]

Thanks for the report.

Just a digression FYI on the English fallback. The English language files are actually not used, Davide just created these files to make translation possible until we moved to a system like TranslateWiki. When a localizable string is used in MathJax, we use both the string id and the English version. This was to ensure that the English fallback is always available (for example, even when downloading the translation data fails or is not finished) and also because Davide wanted to see the explicit English string in the source code. So by design, the English fallback can always happen when something gets wrong and in particular when the language is not available. Also, this should really be en-US since we are using words like "color" rather than "colour".

Regarding the implementation of a general fallback, I remember that there are a HUGE number of languages when I studied the HTML5 validator. So I'm wondering if (arguably naive) fallback implementation like the following could help: when one specify "xx-yy-zz" try it first, next try "xx-yy", next try "xx" and finally the English fallback? I realize that the case of "en-US" and "pt-bz" that exist before "en" and "pt" show that it won't work in general. So perhaps this can be combined with another non-automated method: just specify some mappings in the MathJax.Localization structure that will tell e.g. to use "pt-bz" when "pt" is asked and could be configured as wanted (perhaps via a function setLanguageFallback or something). That way, we won't need to add language data or redirection for all the existing languages, only those that are needed.

[fred-wang]

So I've just added a "remap" entry in the default language data in order to specify an existing fallback language. For example

      pt: {menuTitle: "portugus\u00EA", remap: "pt-br"},
     "pt-br": {menuTitle: "portugu\u00EAs do Brasil"} 

will tell MathJax to use "pt-br" as a fallback for "pt". The remap is computed by the getLocale function. If necessary, this could do something more clever as the heuristic I mentioned above. But I think it's ok for now.

[dpvc]

I think your idea of an automated fallback algorithm makes a lot of sense, especially when combined with the remapping that you already have (which can be used to move from less specific to more specific, as with pt -> pt-br).

I would also suggest allowing something like

pt: {remap: "pt-br"}

so that you can remap a locale without having it specifically in the MathJax menu. (The createLocaleMenu method in MathMenu.js would need to be modified to account for that.) Not that I'm saying to do that for "pt", but just to allow a generic remapping facility that isn't directly tied to the menu.

So I would say go ahead and implement your scheme. That will allow

<script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=...&locale=en-US"></script>

to use the "en" locale (the default), or locale=pt-XX for other XX to first go to pt, which remaps to pt-br.

[dpvc]

Your issue495 branch adds a getLocale function, but I think you can fold that into the setLocale method. Currently, the locale must be looked up every time a string is needed (e.g., for every "Processing ... " message), but it really only needs to be done once, when the locale is first determined. The MathJax.Localization.locale is not the one that controls the menu selection, so it is OK to set this to the actual locale used, even if it is remapped. The menu will still show the one selected by the user. It will be more important once you do the algorithmic lookup, as that would make getLocale more expensive.

[fred-wang]

I've created a new branch issue492.

=> Ready for review.

[fred-wang]

=> UnitTest wanted

[dpvc]

I did a little more testing, and there are two lines in MathJax.js that need to be changed to call MathJax.Localization.setLocale() rather than setting MathJax.Localization.locale directly.

Line 2300 needs to be

MathJax.Localization.setLocale(this.params.locale);

and line 2379 needs to be

{MathJax.Localization.setLocale(config.menuSettings.locale)}

The one difference is that if MathMenu.js is already loaded, this will queue the loading of the locale's MathMenu domain file. I don't think this will be a problem, but it is something to keep in mind for future bug reports.

Alternatively, you could make a copy of setLocale and rename it something like resetLocale then remove the last line about loading the MathMenu domain. Then make setLocale call resetLocale and then do the loading of the domain. Finally, use resetLocale for the two lines above. (Whew!)

[fred-wang]

I added tests for setLocale and &locale=

API/Localization/param-1.html
API/Localization/setLocale-1.html

=> In testsuite

[dpvc]

==> Merged

[fred-wang]

Currently, the fallback is applied when the whole translation for a language is missing. I think that now that "pt" is partly translated, it won't fallback to "pt-bz" but uses "pt" for strings that are translated and "en" for the untranslated strings.

[he7d3r]

That is not good :-(

[pkra]

@helder-git do you have another suggestion?

[dpvc]

@fred-wang, could we copy any missing entries in the pt file from the pt-brfile when the pt files are built, so that they contain no untranslated strings? That is, do the defaulting to pt-br up front when we create the pt files from the TranslateWiki data?

Alternatively, we could not include pt until all the strings are available (so that MathJax selects pt-br when pt is requested).

[he7d3r]

Isn't possible to have something like

englishDefault = { msgKey1: msgText1InEnglish, ... }
fallback1 = { msgKey1: msgText1InFallback1, ... }
//...
fallbackN = { msgKey1: msgText1InFallbackN, ... }
userLanguage = { msgKey1: msgText1InUserLanguage, ... }

and then make something like

$.extend( englishDefault, fallbackN, ..., fallback1, userLanguage )

? If not, @dpvc's first suggestion seems good for now.