Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement language fallback #492

Closed
mattflaschen opened this issue Jun 6, 2013 · 14 comments

Comments

Projects
None yet
5 participants
@mattflaschen
Copy link

commented Jun 6, 2013

For instance, if the code requests a locale of de-CH, it should fallback to de, since currently there is no de-CH. This is an IETF language tag, an extension of ISO 639-1.

That way, you can just request any language without knowing what's implemented. And if a de-CH localization is added later, it will just work a little better, without the client code changing anything.

Fallback to English already works for when you request something like zh which isn't implemented at all.

You may be able to import the fallback configuration from MediaWiki. They have fallbacks configured (e.g. https://git.wikimedia.org/blob/mediawiki%2Fcore.git/d7a61d72231861755f0ccdaf80d7b4388cebd5df/languages%2Fmessages%2FMessagesMwl.php has a fallback to 'pt'), except when the fallback is to English.

Note that they are often circular, though ('pt' falls back to 'pt-br', which falls back to 'pt').

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2013

Thanks for the report.

Just a digression FYI on the English fallback. The English language files are actually not used, Davide just created these files to make translation possible until we moved to a system like TranslateWiki. When a localizable string is used in MathJax, we use both the string id and the English version. This was to ensure that the English fallback is always available (for example, even when downloading the translation data fails or is not finished) and also because Davide wanted to see the explicit English string in the source code. So by design, the English fallback can always happen when something gets wrong and in particular when the language is not available. Also, this should really be en-US since we are using words like "color" rather than "colour".

Regarding the implementation of a general fallback, I remember that there are a HUGE number of languages when I studied the HTML5 validator. So I'm wondering if (arguably naive) fallback implementation like the following could help: when one specify "xx-yy-zz" try it first, next try "xx-yy", next try "xx" and finally the English fallback? I realize that the case of "en-US" and "pt-bz" that exist before "en" and "pt" show that it won't work in general. So perhaps this can be combined with another non-automated method: just specify some mappings in the MathJax.Localization structure that will tell e.g. to use "pt-bz" when "pt" is asked and could be configured as wanted (perhaps via a function setLanguageFallback or something). That way, we won't need to add language data or redirection for all the existing languages, only those that are needed.

@ghost ghost assigned fred-wang Jun 17, 2013

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2013

So I've just added a "remap" entry in the default language data in order to specify an existing fallback language. For example

      pt: {menuTitle: "portugus\u00EA", remap: "pt-br"},
     "pt-br": {menuTitle: "portugu\u00EAs do Brasil"} 

will tell MathJax to use "pt-br" as a fallback for "pt". The remap is computed by the getLocale function. If necessary, this could do something more clever as the heuristic I mentioned above. But I think it's ok for now.

@dpvc

This comment has been minimized.

Copy link
Member

commented Aug 1, 2013

I think your idea of an automated fallback algorithm makes a lot of sense, especially when combined with the remapping that you already have (which can be used to move from less specific to more specific, as with pt -> pt-br).

I would also suggest allowing something like

pt: {remap: "pt-br"}

so that you can remap a locale without having it specifically in the MathJax menu. (The createLocaleMenu method in MathMenu.js would need to be modified to account for that.) Not that I'm saying to do that for "pt", but just to allow a generic remapping facility that isn't directly tied to the menu.

So I would say go ahead and implement your scheme. That will allow

<script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=...&locale=en-US"></script>

to use the "en" locale (the default), or locale=pt-XX for other XX to first go to pt, which remaps to pt-br.

@dpvc

This comment has been minimized.

Copy link
Member

commented Aug 1, 2013

Your issue495 branch adds a getLocale function, but I think you can fold that into the setLocale method. Currently, the locale must be looked up every time a string is needed (e.g., for every "Processing ... " message), but it really only needs to be done once, when the locale is first determined. The MathJax.Localization.locale is not the one that controls the menu selection, so it is OK to set this to the actual locale used, even if it is remapped. The menu will still show the one selected by the user. It will be more important once you do the algorithmic lookup, as that would make getLocale more expensive.

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2013

I've created a new branch issue492.

=> Ready for review.

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2013

=> UnitTest wanted

@dpvc

This comment has been minimized.

Copy link
Member

commented Aug 7, 2013

I did a little more testing, and there are two lines in MathJax.js that need to be changed to call MathJax.Localization.setLocale() rather than setting MathJax.Localization.locale directly.

Line 2300 needs to be

MathJax.Localization.setLocale(this.params.locale);

and line 2379 needs to be

{MathJax.Localization.setLocale(config.menuSettings.locale)}

The one difference is that if MathMenu.js is already loaded, this will queue the loading of the locale's MathMenu domain file. I don't think this will be a problem, but it is something to keep in mind for future bug reports.

Alternatively, you could make a copy of setLocale and rename it something like resetLocale then remove the last line about loading the MathMenu domain. Then make setLocale call resetLocale and then do the loading of the domain. Finally, use resetLocale for the two lines above. (Whew!)

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2013

I added tests for setLocale and &locale=

API/Localization/param-1.html
API/Localization/setLocale-1.html

=> In testsuite

dpvc pushed a commit that referenced this issue Aug 8, 2013

@dpvc

This comment has been minimized.

Copy link
Member

commented Aug 8, 2013

==> Merged

@dpvc dpvc closed this Aug 8, 2013

dpvc pushed a commit that referenced this issue Aug 8, 2013

@dpvc dpvc referenced this issue Aug 8, 2013

Merged

Issue529 #540

@fred-wang

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2013

Currently, the fallback is applied when the whole translation for a language is missing. I think that now that "pt" is partly translated, it won't fallback to "pt-bz" but uses "pt" for strings that are translated and "en" for the untranslated strings.

@he7d3r

This comment has been minimized.

Copy link

commented Oct 8, 2013

That is not good :-(

@pkra

This comment has been minimized.

Copy link
Member

commented Oct 8, 2013

@helder-git do you have another suggestion?

@dpvc

This comment has been minimized.

Copy link
Member

commented Oct 8, 2013

@fred-wang, could we copy any missing entries in the pt file from the pt-brfile when the pt files are built, so that they contain no untranslated strings? That is, do the defaulting to pt-br up front when we create the pt files from the TranslateWiki data?

Alternatively, we could not include pt until all the strings are available (so that MathJax selects pt-br when pt is requested).

@he7d3r

This comment has been minimized.

Copy link

commented Oct 8, 2013

Isn't possible to have something like

englishDefault = { msgKey1: msgText1InEnglish, ... }
fallback1 = { msgKey1: msgText1InFallback1, ... }
//...
fallbackN = { msgKey1: msgText1InFallbackN, ... }
userLanguage = { msgKey1: msgText1InUserLanguage, ... }

and then make something like

$.extend( englishDefault, fallbackN, ..., fallback1, userLanguage )

? If not, @dpvc's first suggestion seems good for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.