Reintroduce pseudo-localization

[zbraniecki]

Coming back from the Unicode Conference, there was a lot of chatter about pseudo-locales.

Fluent already had a pretty good support for pseudo-locales in the past and due to our client-side mode, we offer an exciting approach to pseudo-locales - runtime pseudolocalization.

I'd like to bring back this: https://github.com/l20n/l20n.js/blob/v3.x/src/lib/pseudo.js to modern fluent.

@stasm - do you have any thoughts on how would you like it to work?

[zbraniecki]

Maybe just introducing some generic "post-processing" on messages in MessageContext, and then adding fluent-pseudolocale would work as the first step?

let ctx = new MessageContext(['ar'], {
  process: fluent_pseudolocales.transform.bind('ar-XB')
});
let msg = ctx.formatValue('l10n-id');

[stasm]

Maybe just introducing some generic "post-processing" on messages in MessageContext, and then adding fluent-pseudolocale would work as the first step?

That was my first thought as well. A few additional thoughts below. I'll try to have answers tomorrow.

We have to take into account how this will interact with the language negotiation. Would we expect the user to set their requested locale to a pseudolocale in order to enable it? Would we require that developers add pseudolocales to the list of available locales in their app?

Perhaps it would make sense to encode pseudolocales as Unicode extensions to BCP47? Something like ab-CD-u-pseudo-accent or ab-CD-u-pseudo-rtl. The language negotiation process would then still correctly pick the regular ab-CD for fetching translation resources. Some logic would then be responsible for transforming the fetched resource using the fluent-pseudo module.

What should be the outcome of formatting a date or a number in a pseudolocalized translation?

Also, the first step might be to only support build-time pseudolocalization.

[zbraniecki]

Google just went for en-XA, and ar-XB and added them to CLDR. So we can get internationalized date/time from CLDR 31 if we use those two.

Now, my problem with this is that because they used en-XA and not fr-XA, the numbers still look the same. I recommended them fr-XA, but it may be too late.

Since we'd be doing runtime pseudo, maybe we don't need extensions (and they wouldn't be unicode extensions, but rather variants ( Google originally used en-psaccent and ar-psaccentrtl or sth like that).

Maybe all we need is:

let ctx = new MessageContext('pl', {
  process: pseudo
});

and it'll transform polish strings? This way we could get a pseudo of the current locale, irrelevant of what it is.

[stasm]

How would you decide when to turn pseudolocalization on? A different logic independent of the language negotiation?

[zbraniecki]

How would you decide when to turn pseudolocalization on? A different logic independent of the language negotiation?

Yeah! This way the user can either detect pseudo from a langtag (oh, you're using XA region?) or by some checkbox (show me pseudolocale).
Since we're on client-side at runtime, that would mean no rebuilding, restarting or anything. Just take the exact locale we use, whatever it is, and recompute for pseudo.

[stasm]

Yeah! This way the user can either detect pseudo from a langtag (oh, you're using XA region?) or by some checkbox.

If the user sets their requested to en-XA and the available only have en-US, the result of the language negotiation will be `en-US`. At the moment when we’d create the MessageContext we wouldn’t know the region was XA. On the other hand, if we add en-XA to the list of available locales, and the files for it do not exists on disk, we will fail to fetch anything. We’d need to extend the IO logic to fetch them. This might mean moving the pseudolocalization to fluent-web. I think it would be better to have it on a lower level though. Does the language negotiation preserve extensions found on the requested locales?

[zbraniecki]

Oh, you're right.

I think it would be better to have it on a lower level though.

I agree.

Does the language negotiation preserve extensions found on the requested locales?

yes.

So, maybe private-extension? fr-FR-x-pseudo ?
We just need to make sure that if we see a pseudo like this, we actually feed en-XA, ar-XB to Intl API (so that CLDR picks it up)

[stasm]

There's discussion in http://unicode.org/cldr/trac/ticket/3971 and http://unicode.org/cldr/trac/ticket/9819 on why CLDR didn't go for variant tags. It's mostly about compatibility with existing code. Also, since en-XA and ar-XB are now in CLDR we should stick to these codes. I wish we hadn't missed the discussion when it happened.

I'm reconsidering my stance on where this logic should live. Having it higher up, e.g. in fluent-web would allow multiple approaches:

• build-time transformation,
• AST transformation right after IO,
• string transformation after ctx.format().

fluent-web can also transform translations in a way which preserves HTML for the overlay mechanic.

[Pike]

I'd think that the most accute way to implement the actual pseudo localization would be on the AST?

[stasm]

On buildtime or on runtime?

[Pike]

For both, I guess.

[stasm]

Transforming the runtime AST means doing the transformation inside of MessageContext. That still might a viable option given my earlier comments: fluent-web could supply a markup-aware transform function to the MessageContext constructor. Compared to transforming the result of ctx.format(), this would have the advantage of only transforming TextElements in the translation rather than the whole string.

We still need to solve the problem of fetching valid locale files. Given that it looks like fluent-web (or fluent-react) would need to handle the pseudolocalization anyways (if only to be HTML-aware), I think it makes sense to special-case en-XA and ar-XB in their IO.

For example, given the following result of language negotiation:

• requested: en-XA, de
• available: en-XA, en-US, de
• default: en-US
• negotiated: en-XA, de, en-US

…a developer using fluent-react will need to add a special case to the IO code which fetches en-US when en-XA is requested. This sounds okay to me since the same developer has already put en-XA among the available locales.

let ctx = new MessageContext(negotiated, { pseudo: makeAccent });
ctx.addMessages( /* en-US translations to be transformed into en-XA */);

Or, if the build pipeline is capable of building pseudolocales up front, the IO code would simply fetch the pre-made en-XA files.

let ctx = new MessageContext(negotiated);
ctx.addMessages( /* en-XA  translations generated on build-time */);

[zbraniecki]

I do not agree with Stas that we have to use en-XA and ar-XB here. I believe it's perfectly fine for us to use whatever mechanism we want to use to recognize pseudolocales, and then just make sure to collapse in Intl constructor onto en-XA and ar-XB for Intl API / CLDR.

[stasm]

Note that the approach from my previous comment will work with any scheme of specifying pseudolocales. In my example I chose to put en-XA in requested but it could also be an app-specific pref which handles that. This is also how I understood your comment from 5 days ago.

There's value in using en-XA, ar-XB now that they were standardized in the CLDR. They will become recognizable names for pseudolocales and with time will gain support in various tools and platforms.

[zbraniecki]

@stasm - would you have time to draft a plan to get this into a POC state? I'm happy to commit to work on that, but would prefer to follow your vision.