i18n: placeholders #9114

exterkamp · 2019-06-04T00:37:35Z

Add $placeholder$ syntax to i18n. This will allow us to escape all [mark](down) and {ICU} syntax so that it cannot be translated! Check out the README.md for a good summary of the new process.

Tested with tc/ and we are looking pretty good afaict.

Additional fun:

No-op to locales/ files

Follow ups:

multi-line descriptions
~~@description required~~
type getMessageInstanceIdFn param values with Record<string, string | number>
~~make test to fail when adding new @meanings~~

Related Issues/PRs
More talk: #6945
Evidence of need: #9197

brendankenny

only took a quick look, but here are some first thoughts.

Should also do a string with a plural replacement and a {timeInMs, number, milliseconds} replacement to explore how those could be handled

brendankenny · 2019-06-04T01:03:34Z

lighthouse-core/audits/byte-efficiency/unminified-css.js

+        content: '[',
+      },
+      link_end: {
+        content: '](https://developers.google.com/web/tools/lighthouse/audits/minify-css)',


this is a pretty unfortunate syntax. Maybe we want to come up with something that expands to this in collect-strings.js?

Maybe we want to come up with something that expands to this in collect-strings.js?

Just to be clear, I meant this as a super low priority idea that we can think about while we do everything else first and implement it later if there's a nice fix. It kind of sucks to look at/edit but we could live with it if need be (we may need a test to make sure a typo in one of the contents doesn't accidentally break a link)

That was also my thought. This is super heavyweight to have in each .js file. I can try to experiment.

something that expands to this in collect-strings.js

Let's just keep it as [some text {name} or even placeholders](url) and expand it to additional placeholders in collect strings.

Like auto convert [ and ](url) into link_start link_end in collect strings?

I worry about making special cases, adding unseen complexity behind the scenes. I think part of the positives of a change like this is that it removes all ambiguity.

yeah - I think that'd be dope

brendankenny · 2019-06-04T01:07:47Z

lighthouse-core/lib/i18n/i18n.js

+      const key = entry[0];
+      const value = entry[1];
+      //use key and value here
+      let regexStr = `\$${key}\$`;


I think we'll definitely want a collect-strings test that verifies that all the placeholders are in the string (would we also want the stricter check that there aren't any escaped values that don't have placeholders?)

brendankenny · 2019-06-04T01:09:50Z

lighthouse-core/lib/i18n/i18n.js

+      //use key and value here
+      let regexStr = `\$${key}\$`;
+      console.log(key, value, regexStr);
+      localeMessage = localeMessage.replace(regexStr, value.content);


maybe to avoid the issue with positional replacements ($1 etc), we could use use the named replacements in values if value.content is \$\d+ or whatever? We'll just have to figure out how to do the mapping (just use key?)

exterkamp · 2019-06-04T19:20:36Z

lighthouse-core/audits/byte-efficiency/unminified-css.js

+    'other {# errors found}}',
+  /** [ICU Syntax] Some gendered (ICU select) explanation... */
+  explanationGender: {
+    message: 'Someone minified this, {direct_replace_name}. {static_replacement} {person, select, ' +


This contains a 1. direct replacement in {ICU string replace} a static_replacement that is covered by a placeholder, and "Google valid" ICU select syntax.

exterkamp · 2019-06-04T19:22:17Z

lighthouse-core/audits/byte-efficiency/unminified-css.js

+  },
+  /** [ICU Syntax] Some gendered (ICU select) explanation... */
+  explanationGender2: {
+    message: 'Someone minified this, {name}. {static_replacement} {person, select, ' +


This example uses placeholders to allow us to make more perfectly formed translation strings, but stutters with {name} in the message, and the content being the same {name} since that is the replacement we want to fill it with. The advantage here is that we can give the translators an example value of the replacement, e.g. here it is a name, so the example like "Karen" would give more context to the translators.

exterkamp · 2019-06-04T20:43:27Z

Started some docs on explaining this for future development

exterkamp · 2019-06-12T15:13:31Z

Update: Having a lot of trouble on the backend infra side converting these .json messages into .xmb's and roundtripping them back to .json intact.

exterkamp · 2019-06-12T20:07:01Z

Okay, fixed the roundtrip issues 👍 But design question @paulirish @brendankenny @patrickhulce @hoten :
The intial messages.json format in en-US looks like this (fake-example):

  "lighthouse-core/audits/accessibility/accesskeys.js | failureTitle": {
    "message": "$ACCESS_KEY$ $VALUES$ are not unique",
    "placeholders": {
      "ACCESS_KEY": {
        "content": "`[accesskey]`"
      },
      "VALUES": {
        "example": "12",
        "content": "{value, number}"
      }
    },
    "description": "Title of an accesibility audit that evaluates if the ARIA HTML attributes are misaligned with the aria-role HTML attribute specificed on the element, such mismatches are invalid. This title is descriptive of the failing state and is shown to users when there is a failure that needs to be addressed."
  },

And after a roundtrip it looks like this (in another language here its en-XA):

  "lighthouse-core/audits/accessibility/accesskeys.js | failureTitle": {
    "message": "$ACCESS_KEY$ $VALUES$ åŕé ñöţ ûñîqûé",
    "placeholders": {
      "ACCESS_KEY": {
        "content": "`[accesskey]`"
      },
      "VALUES": {
        "content": "{value, number}"
      }
    }
  },

Notice the small variations, the roundtrip loses

examples
descriptions

Which is desired since they aren't really needed post translation.

But it also brings along all the static placeholder text. Which is a lot of extra bytes. Each language will have all the placeholder data with it. Thats a lot of copies of the strings. Suggestion:

Keep all the strings, it makes each locale.json file totally self-contained
Make a backfill mechanism to fallback to en-US to lookup the contents.

Thoughts?

connorjclark · 2019-06-13T23:18:28Z

But it also brings along all the static placeholder text. Which is a lot of extra bytes. Each language will have all the placeholder data with it. Thats a lot of copies of the strings. Suggestion:

could collect-strings save off a placeholders.json for just the icuMessageId -> placeholders data here, and just load placeholders.json here? Then delete placeholders from each locale file. I don't think having them self contained is important.

But these extra bytes don't end up in the report (just the bundle), so maybe it's not worth the effort.

…fter :(

brendankenny

last round! Looks great

lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js

lighthouse-core/lib/i18n/README.md

lighthouse-core/lib/i18n/i18n.js

brendankenny · 2019-07-23T18:27:04Z

lighthouse-core/scripts/i18n/collect-strings.js

+}
+
+/**
+ * Take a series of messages and apply ĥât̂ markers to the translatable portions


👍 👍 🎩

lighthouse-core/scripts/i18n/collect-strings.js

brendankenny · 2019-07-23T18:44:50Z

lighthouse-core/scripts/i18n/collect-strings.js

-        psuedoLocalizedString.push(useHatForAccentMark ? `\u0302` : `\u0301`);
-        useHatForAccentMark = !useHatForAccentMark;
+            // Stop empty placeholders from being added to every message
+            if (Object.entries(converted.placeholders).length > 0) {


you could move this into convertMessageToPlaceholders, which would keep it encapsulated there (they just come out with no placeholders prop if there are no placeholders) and allow you to do the tests as you were in 125cb1d

something like this should be fine:

if (Object.entries(icuDefn.placeholders).length > 0) { icuDefn.placeholders = undefined; }

then in here it would be

/** @type {ICUMessageDefn} */ const icuDefn = { message: converted.message, description, placeholders: converted.placeholders, };

and while that's technically copying over undefined to the new object, this is bound for json which will strip out undefined values anyways

Added, but I had to add a //@ts-ignore dunno if that was intended 😮 Unless you want to make placeholders | undefined in ts.

Added, but I had to add a //@ts-ignore dunno if that was intended 😮 Unless you want to make placeholders | undefined in ts.

yeah, this is terrible it had to do that, but I wanted to do a follow up cleaning up some of these types, so I can handle it there.

brendankenny · 2019-07-23T23:38:09Z

needs a rebase, too

brendankenny

LGTM!

🎉 🎉

exterkamp added 3 commits June 3, 2019 13:23

basic format

656acce

Basic placeholder syntax.

fc762a5

minor corrections

d1cdeab

googlebot added the cla: yes label Jun 4, 2019

exterkamp added the i18n internationalization thangs label Jun 4, 2019

brendankenny mentioned this pull request Jun 4, 2019

Unescaped HTML in translation text #6945

Closed

brendankenny reviewed Jun 4, 2019

View reviewed changes

iterating

74e11c1

vercel bot deployed to staging June 4, 2019 18:42 View deployment

Added ICU plural and ICU select examples.

981e028

vercel bot deployed to staging June 4, 2019 19:00 View deployment

Gender example has replacement AND ICU.

6d2dd5d

vercel bot deployed to staging June 4, 2019 19:09 View deployment

less stutter syntax

f22ee1b

vercel bot deployed to staging June 4, 2019 19:18 View deployment

exterkamp commented Jun 4, 2019

View reviewed changes

docs

c66c0c6

vercel bot deployed to staging June 4, 2019 20:42 View deployment

paulirish mentioned this pull request Jun 6, 2019

Internationalization of Lighthouse (Phase 2) #7238

Closed

36 tasks

Updated README.

70a5a07

vercel bot deployed to staging June 12, 2019 20:18 View deployment

exterkamp mentioned this pull request Jun 12, 2019

i18n: enable local debug locale, en-XL #9192

Merged

correct-strings

a7fc83a

vercel bot deployed to staging July 3, 2019 07:00 View deployment

updated i18n.js

0de5485

vercel bot deployed to staging July 22, 2019 21:48 View deployment

updated columnName description

b95b96b

vercel bot deployed to staging July 22, 2019 21:57 View deployment

actually can't assert that in the tests. The undefined part is done a…

90efcff

…fter :(

vercel bot deployed to staging July 22, 2019 23:08 View deployment

@example {example} varName

6e6ef24

vercel bot deployed to staging July 23, 2019 01:30 View deployment

whoops. console log left in

6a9ef28

vercel bot deployed to staging July 23, 2019 14:58 View deployment

brendankenny reviewed Jul 23, 2019

View reviewed changes

brendankenny mentioned this pull request Jul 23, 2019

core(tbt): Clean up total blocking time code #9409

Merged

brendan feedback

7762381

vercel bot deployed to staging July 23, 2019 23:05 View deployment

Merge branch 'master' into placeholders

6c19a17

vercel bot deployed to staging July 24, 2019 00:09 View deployment

Add sanity check for ctc files

be34f7e

vercel bot deployed to staging July 24, 2019 00:34 View deployment

I feel like my linter is turned off

23a68d5

vercel bot deployed to staging July 24, 2019 00:37 View deployment

Merge branch 'master' into placeholders

3685dd6

vercel bot deployed to staging July 24, 2019 06:53 View deployment

brendankenny approved these changes Jul 24, 2019

View reviewed changes

brendankenny merged commit 94a6e38 into master Jul 24, 2019

brendankenny deleted the placeholders branch July 24, 2019 22:13

This was referenced Jul 25, 2019

i18n: align type names with docs #9461

Merged

i18n: use tsc to collect UIStrings #9487

Merged

exterkamp mentioned this pull request Aug 1, 2019

I18N: Missing translations for audits #9492

Closed

paulirish pushed a commit that referenced this pull request Nov 6, 2019

i18n: expand placeholder support (#9114)

5696f1e

snyk-bot mentioned this pull request Mar 21, 2020

[Snyk] Upgrade lighthouse from 5.1.0 to 5.6.0 godaddy/lighthouse4u#13

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i18n: placeholders #9114

i18n: placeholders #9114

exterkamp commented Jun 4, 2019 •

edited

Loading

brendankenny left a comment

brendankenny Jun 4, 2019

brendankenny Jun 4, 2019

exterkamp Jun 4, 2019

connorjclark Jun 4, 2019 •

edited

Loading

exterkamp Jun 4, 2019 •

edited

Loading

connorjclark Jun 4, 2019

brendankenny Jun 4, 2019

brendankenny Jun 4, 2019

exterkamp Jun 4, 2019

exterkamp Jun 4, 2019

exterkamp commented Jun 4, 2019

exterkamp commented Jun 12, 2019

exterkamp commented Jun 12, 2019

connorjclark commented Jun 13, 2019

brendankenny left a comment

brendankenny Jul 23, 2019

brendankenny Jul 23, 2019

exterkamp Jul 23, 2019 •

edited

Loading

brendankenny Jul 23, 2019

brendankenny commented Jul 23, 2019

brendankenny left a comment

i18n: placeholders #9114

i18n: placeholders #9114

Conversation

exterkamp commented Jun 4, 2019 • edited Loading

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark Jun 4, 2019 • edited Loading

Choose a reason for hiding this comment

exterkamp Jun 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exterkamp commented Jun 4, 2019

exterkamp commented Jun 12, 2019

exterkamp commented Jun 12, 2019

connorjclark commented Jun 13, 2019

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exterkamp Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Jul 23, 2019

brendankenny left a comment

Choose a reason for hiding this comment

exterkamp commented Jun 4, 2019 •

edited

Loading

connorjclark Jun 4, 2019 •

edited

Loading

exterkamp Jun 4, 2019 •

edited

Loading

exterkamp Jul 23, 2019 •

edited

Loading