x/text: localization support

[mpvl]

This issue is intended as an umbrella tracking issue for localization support.

Localization support includes:

• formatting values, such as numbers, currencies, and dates in a language- and region-specific way.
• marking text to be translated in fmt-ed text and templates.
• integration into translation pipeline

Details to be covered in design docs.

[mpvl]

Added design doc: https://github.com/golang/proposal/blob/master/design/12750-localization.md

[maximilien]

Here is one solution: https://github.com/maximilien/i18n4go

[mpvl]

@maximilien: i18n4go does not address localized formatting of values like numbers and I think it will be difficult to retrofit it properly. In case of selecting translation variants based on the linguistic features of the arguments, you'll end up with the same struggle one witnesses with localization frameworks for other languages.
Also, i18n4go extracts all strings and then uses an exclusion file. This may work well for command line tools or applications where most strings need localization, but this is not the norm. It breaks down when a large number of the strings in code do not need to be localized. For example, internal error messages are often not localized and may actually be the bulk of the text.
Addressing both issues will likely result in a different API, for example like the one proposed. The implementation of the proposed API is more complex, but it eliminates the need to generate a parallel version of the code and T wrappers.

This proposal is fairly agnostic about translation pipelines, though. So it may be possible to fit this proposal on top of the i18n4go translation pipeline. Seems like a convenient first target.

[infogulch]

Using the Printf of message.Printer has the following consequences:

• ...
• the format string is now a key used for looking up translations

Is the format string by itself sufficient for determining the context? I can imagine a very simple Printf used like m.Printf("%s: %d", m, i) where the format string %s: %d could appear a dozen times throughout a codebase with very different contexts. (You could argue that this is a very poor format string to begin with, but it still demonstrates my concern.)

I must admit I'm not very familiar with localization problems and this may not be an issue in practice.

[mpvl]

@infogulch It is indeed not enough. In my provisionally worked out API I do define a Key function that can be used for things like adding meaning and alternatives. I left it out of the design doc to not go into details too much. (I also stripped about 1/3rd of my original draft; maybe I went a bit overboard.)

Note that as the string has no meaning in itself, you could always write the format string as, for example, "Archive (verb)" and "Archive (noun)" and supply a "translation" for these in English ("Archive" for both). But this does not address all concerns. A more general solution:

Printf would have the following signature:

func (p *Printer) Printf(key Reference, args ...interface{}) (n int, err error) {

where Reference is either a string or a result from a func like

func Key(id string, fallback ...string) Reference {

This allows the familiar Printf() usage while addressing the concerns you raised. Many localization frameworks have a solution of a similar nature.

But the example string you provide does raise another good point: there may be format strings one does not want to translate at all while still using the message package to substitute localized values. This is possible as is (e.g. fmt.Printf("%s: %d", m.Print(m), m.Print(i))), but may be a bit clunky. A bit better may be something like m.Printf(message.Raw("%s: %d"), m, i), where the use of Raw makes extraction skip the string. I don't think there are too many cases where this is used, though. Even "%s: %d" will vary per language. But single-value substitutions like "%2.3d" should probably be excluded from translation.

[akavel]

As far as plurals are concerned, I've seen some elaborate examples, but what somewhat skimming the doc, it seems they can use only "<", ">", and "=" operators; I didn't read it 100% thoroughly however, so I may be wrong. I'll thus let myself ask here for clarifications: are the proposed mechanisms enough to cater for the rule for e.g. Polish language? In a version I found on Weblate site, it's described as [1] [2]:

n==1 ? 0 :                                              // "single"
 n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 :    // "few"
 2                                                      // "many"

which seems to me quite fine, correctly giving e.g.:

1 orangutan
2-4 orangutany
5-9 orangutanów
10, 11, 12, ..., 21 orangutanów
22 orangutany
101 orangutanów
102 orangutany
etc.

[mpvl]

@akavel: one should distinguish selectors from the rules you mention. The rules you refer to (which are defined in CLDR) would be used by the plural package to map numbers to a small set of plural categories (in the case of your example: single, few and many). The selectors subsequently pick alternatives based on these simplified categories. The maximum number of such categories, IIRC, is 6 (e.g. for Arabic).
Most localization frameworks that support plural, allow selecting on these categories only. ICU adds selecting on the number value (using "="). The matching algorithm defined in this proposal is a bit different from ICU, allowing also for Vars and selecting on "<" and ">". The selectors will often be generated or written by translators (with the help of a GUI) so they should remain simple.

In my foreseen implementation, it is really up to the feature implementation to interpret selectors. This means that there is a lot of flexibility in supporting wild feature value matching. However, if one looks at linguistic grammars like LFG and HSPG, which use many more features, the set of possible feature values is usually small.

The doc is indeed a bit sparse here (as well as all other topics, really).

[maximilien]

@mpvl, sounds good. Happy to try and integrate once you have something ready to try. Best.

[abourget]

Have you guys seen this one? https://github.com/nicksnyder/go-i18n seems pretty solid at first sight.

[abourget]

It uses JSON as its base format, has tooling to help with the translation workflows

[BenLubar]

By the way, I submitted some formatting fixes for the proposal doc a few weeks ago.

https://go-review.googlesource.com/19753

Not sure what I was supposed to do to get it reviewed.