core(i18n): add locale fallbacks when language not supported #5746

brendankenny · 2018-07-28T03:49:42Z

Sets up locale fallbacks, using the pretty common method of pruning on the last - until some prefix of the original locale is found, falling back to the default locale if no match is found.

Also adds a test to make sure that for any locale we set we also have a fallback for that language. So if e.g. we had pt-BR but not pt-PT for some reason, we'll be guaranteed to have pt to fall back to. Looking at our (eventual) list of supported languages, we shouldn't have to do any artificial mappings to support this like I'm doing in this PR.

Since locale is part of config.settings, it makes sense to initialize and check it in Config, so this moves it there. This allows removing some of the optional uses that had defaults coming from a few different places depending on the execution path taken.

simplify default locale control flow by centralizing in Config

brendankenny · 2018-07-28T03:51:39Z

lighthouse-core/lib/i18n.js

+ * @param {string=} locale
+ * @return {LH.Locale}
+ */
+function lookupLocale(locale = MessageFormat.defaultLocale) {


MessageFormat.defaultLocale is actually en when no locale info is found, not en-US. Not sure if we have an issue with that since we assume it's en-US everywhere else. As a result, we'll only get the en-US default set at the end of this function if a specific locale we don't support is requested. Do we want to change the last line to return en instead? Or just live with this

these words wash over me like an ocean wave. 🌊 i don't know what hit me but it was powerful and disorienting.

😕

these words wash over me like an ocean wave

haha. Basically everywhere we pretend the default has been en-US, but the current i18n.getDefaultLocale has been trying to return en since that's what the default fallback of MessageFormat.defaultLocale is. It just then finds out there is no en strings file, so the function falls back to the hardcoded en-US. Once we have an en strings file, that will change.

So,

if you request whatever the default is for where LH is running, you'll get en

if you request a locale that we don't have (and have no fallback for), you'll get en-US

I'm wondering if we should

change default fallback everywhere to en to match MessageFormat

change MessageFormat.defaultLocale so that if it returns en, to be en-US and not change everywhere else

live with the inconsistency

brendankenny · 2018-07-28T03:52:23Z

lighthouse-core/lib/i18n.js

@@ -118,7 +134,7 @@ const _icuMessageInstanceMap = new Map();
 * @return {{formattedString: string, icuMessage: string}}
 */
 function _formatIcuMessage(locale, icuMessageId, icuMessage, values) {
-  const localeMessages = LOCALES[locale] || {};
+  const localeMessages = LOCALES[locale];


locale is always a LH.Locale, so standing behind our type checking here by having no fallback :)

brendankenny · 2018-07-28T03:55:02Z

lighthouse-core/lib/locales/index.js

+const locales = {
+  'ar': require('./ar-XB.json'), // TODO: fallback not needed when ar translation available
+  'ar-XB': require('./ar-XB.json'),
+  'en': require('./en-US.json'), // en-* fallback


both ar and en will exist when we get our full languages back. If we want to do other mappings we can do them in here like this or make a simple map in i18n.js internal to lookupLocale()

paulirish · 2018-07-30T19:00:03Z

lighthouse-core/config/config.js


    // Override any applicable settings with CLI flags
    const settingsWithFlags = merge(settingWithDefaults || {}, cleanFlagsForSettings(flags), true);

+    // Use locale specified in flags or settings, allowing i18n system to select fallback if needed.
+    settingsWithFlags.locale = i18n.lookupLocale((flags && flags.locale) || settingsJson.locale);


can you extract (flags && flags.locale) || settingsJson.locale to a const that is defined above L415. between fallbacks and merges i want to be quite clear what is affecting locale and what isnt. (in this case the merge apparently doesnt)

Hmm, I was putting it down here to attempt to make it clear that you can just ignore everything above for locale, this is the place that it's determined and set (and that defaultSettings.locale can't possibly influence the value).

Do you want me to set it on settingWithDefaults or cleanFlagsForSettings(flags) and let the merge merge it in?

paulirish · 2018-07-30T19:01:37Z

lighthouse-core/config/constants.js

@@ -39,7 +39,7 @@ const defaultSettings = {

  // the following settings have no defaults but we still want ensure that `key in settings`
  // in config will work in a typechecked way
-  locale: null, // default determined by the intl library
+  locale: 'en-US', // actual default determined by Config using lib/i18n


since Config.initSettings will never use this, what is it used for

since Config.initSettings will never use this, what is it used for

since we declare the constants in this file to be a valid Config.Settings, unfortunately we have to give some valid value here. I can't think of a good way to not do that while still retaining all the benefits we currently have of the type-checked Settings and making sure every settings property that needs to be defined has a valid value.

paulirish · 2018-07-30T19:02:28Z

lighthouse-core/lib/i18n.js

+ */
+function lookupLocale(locale = MessageFormat.defaultLocale) {
+  // TODO: could do more work to sniff out defaultLocale
+  const canonicalLocale = Intl.getCanonicalLocales(locale)[0];


compat-wise, how do we look here across browsers and node. do we need this one conditionally polyfilled too?

compat-wise, how do we look here across browsers and node. do we need this one conditionally polyfilled too?

it comes with Intl, and it seems like we're assuming at least that exists? The function doesn't check valid tags or need a list of real locales or anything, it's just the capitalization and...maybe some other formatting stuff I can't remember.

Though from https://nodejs.org/api/intl.html it seems like maybe there's no Intl object at all if node was compiled with --with-intl=none?

the mdn compat table is out of date, but it at least indicates support was added later after the Intl basics:

unfortunately the only polyfill appears to be deep inside of Intl.js so hopefully we don't need it?

paulirish · 2018-07-30T19:04:02Z

lighthouse-core/lib/i18n.js

+ * @param {string=} locale
+ * @return {LH.Locale}
+ */
+function lookupLocale(locale = MessageFormat.defaultLocale) {


these words wash over me like an ocean wave. 🌊 i don't know what hit me but it was powerful and disorienting.

😕

paulirish · 2018-07-30T19:11:37Z

lighthouse-core/runner.js

@@ -218,7 +217,7 @@ class Runner {
   */
  static async _runAudit(auditDefn, artifacts, settings, runWarnings) {
    const audit = auditDefn.implementation;
-    const status = `Evaluating: ${i18n.getFormatted(audit.meta.title)}`;
+    const status = `Evaluating: ${i18n.getFormatted(audit.meta.title, 'en-US')}`;


nice. good call.

paulirish · 2018-07-30T19:14:49Z

tsconfig.json

@@ -13,6 +13,7 @@
      "./typings"
    ],

+    "resolveJsonModule": true,


this is for our messages files?

this is for our messages files?

yeah, no reason to keep tsc from not seeing them

core(i18n): add locale fallbacks when language not supported

32e1857

simplify default locale control flow by centralizing in Config

brendankenny requested review from patrickhulce and paulirish as code owners July 28, 2018 03:49

brendankenny commented Jul 28, 2018

View reviewed changes

paulirish mentioned this pull request Jul 28, 2018

Internationalization of Lighthouse (Phase 1) #5719

Closed

24 tasks

brendankenny commented Jul 28, 2018

View reviewed changes

paulirish reviewed Jul 30, 2018

View reviewed changes

brendankenny added 2 commits July 30, 2018 13:22

feedback

a4475e1

more feedback

2c0061e

paulirish approved these changes Jul 30, 2018

View reviewed changes

paulirish merged commit e367395 into master Jul 30, 2018

paulirish deleted the localefallback branch July 30, 2018 21:47

paulirish mentioned this pull request Jul 30, 2018

report(i18n): use LHR locale for toLocaleString #5734

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core(i18n): add locale fallbacks when language not supported #5746

core(i18n): add locale fallbacks when language not supported #5746

brendankenny commented Jul 28, 2018

brendankenny Jul 28, 2018

paulirish Jul 30, 2018

brendankenny Jul 30, 2018

brendankenny Jul 28, 2018

brendankenny Jul 28, 2018

paulirish Jul 30, 2018

brendankenny Jul 30, 2018

paulirish Jul 30, 2018

brendankenny Jul 30, 2018 •

edited

Loading

paulirish Jul 30, 2018

brendankenny Jul 30, 2018

paulirish Jul 30, 2018

paulirish Jul 30, 2018

paulirish Jul 30, 2018

paulirish Jul 30, 2018

brendankenny Jul 30, 2018

core(i18n): add locale fallbacks when language not supported #5746

core(i18n): add locale fallbacks when language not supported #5746

Conversation

brendankenny commented Jul 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny Jul 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny Jul 30, 2018 •

edited

Loading