Selectors docs regarding use of NUMBER are incorrect (and a bad idea)

[spookylukey]

Here:

https://projectfluent.org/fluent/guide/selectors.html

If the translation requires a number to be formatted in a particular non-default manner, the selector should use the same formatting options. The formatted number will then be used to choose the correct CLDR plural category which, for some languages, might be different than the category of the unformatted number:

your-score =
   { NUMBER($score, minimumFractionDigits: 1) ->
       [0.0]   You scored zero points. What happened?
      *[other] You scored { NUMBER($score, minimumFractionDigits: 1) } points.
   }

The text claims that the formatted version of $score will be used to do the matching. In reality this doesn't happen in the reference implementation - see:

• https://github.com/projectfluent/fluent.js/blob/master/fluent/src/resolver.js#L222
• https://github.com/projectfluent/fluent.js/blob/master/fluent/src/types.js#L78 (FluentNumber.match)
• https://github.com/projectfluent/fluent.js/blob/master/fluent/src/types.js#L116 (FluentSymbol.match)

The 0.0 is parsed as a number literal, converted to a FluentNumber, and then compared by value to the FluentNumber instance that results from the first NUMBER call. I added a test to the suite to confirm this, and it makes no difference if you change 0.0 to 0 or 0.00, or if you increase/decrease the value of minimumFractionDigits etc - it is doing a numerical comparison, not taking into account the formatting options.

Further, the behaviour described in the docs would be a bad idea, AFAICS:

1. Number formatting can be quite expensive, we want to avoid it for selector matching.

2. If the formatted number should appear in the variant key expression, this presumably means it needs to be changed by translators e.g. in the example, 0.0 when considered as a string matches the formatted NUMBER expression for English, but it would have to be changed to 0,0 for German etc. I'm guessing translators would have confusion over whether to change these numbers like this.

3. If the formatted number should appear in the variant key expression, all kinds of confusion occurs in terms of parsing/interpreting the FTL. For English locales, the formatted number looks like the number literal for small numbers (1, 2.3 etc.), but not for large numbers (e.g. 1,000 which is not a valid number literal.). For other locales, sometimes the formatted number will again not be a valid number literal, sometimes it might look like a number literal but for a different number (e.g. 1.000 is German for one thousand, but would be parsed as a valid FTL number literal for one).

[stasm]

Thanks. This looks like two issues:

1. The wording in the Guide is misleading. You give valid reasons for why it is so.
2. The fluent.js implementation doesn't currently look at the decimal part of numbers.

Let's use this issue to refine the wording in the Guide. I'll then file a bug to add the desired behavior to fluent.js.

The motivation here is to be able to tell 1 and 1.0 apart. Some languages use a different plural form for these. I think English is one of them, actually: 1 apple but 1.0 apples.

The spec should mention that the decimal part of the number is significant, even if it's all zeros. We shouldn't format the number however, as that might produce non-digits, too, and use different separators. The implementation might actually need to build a custom representation of the number just for this comparison. Does this sound like a reasonable thing to do?

[stasm]

Thanks. This looks like two issues:

1. The wording in the Guide is misleading. You give valid reasons for why it is so.
2. The fluent.js implementation doesn't currently look at the decimal part of numbers.

Let's use this issue to refine the wording in the Guide. I'll then file a bug to add the desired behavior to fluent.js.

The motivation here is to be able to tell 1 and 1.0 apart. Some languages use a different plural form for these. I think English is one of them, actually: 1 apple but 1.0 apples.

The spec should mention that the decimal part of the number is significant, even if it's all zeros. We shouldn't format the number however, as that might produce non-digits, too, and use different separators. The implementation might actually need to build a custom representation of the number just for this comparison. Does this sound like a reasonable thing to do?

[spookylukey]

@stasm - In terms of fixing fluent.js, it seems workable, although I'm wondering where the data would come from for this. Does CLDR even distinguish this case? AFAICS it doesn't. Also it seems like the algorithm for determining a match is already quite complex (and not clearly specified anywhere I can find, other than in the JS implementation), this could be making it even more complex.

[spookylukey]

@stasm - In terms of fixing fluent.js, it seems workable, although I'm wondering where the data would come from for this. Does CLDR even distinguish this case? AFAICS it doesn't. Also it seems like the algorithm for determining a match is already quite complex (and not clearly specified anywhere I can find, other than in the JS implementation), this could be making it even more complex.

[spookylukey]

Actually, thinking some more, I'm confused about the need for this feature or how it should work.

In English, your use case would be covered by:

apples-kilos = There are { NUMBER($qty, minimumFractionDigits: 1) } kilos of apples

i.e. you use the plural form 'kilos' always, because 1 becomes 1.0 which takes the plural. I don't see the need for some special matching functionality for English - but it does require the translator to be aware of this issue.

For other languages I don't know how it would work.

[spookylukey]

Actually, thinking some more, I'm confused about the need for this feature or how it should work.

In English, your use case would be covered by:

apples-kilos = There are { NUMBER($qty, minimumFractionDigits: 1) } kilos of apples

i.e. you use the plural form 'kilos' always, because 1 becomes 1.0 which takes the plural. I don't see the need for some special matching functionality for English - but it does require the translator to be aware of this issue.

For other languages I don't know how it would work.

[zbraniecki]

Does CLDR even distinguish this case? AFAICS it doesn't.

It does. You can read more about it in https://www.unicode.org/reports/tr35/tr35-numbers.html#Language_Plural_Rules

In particular, a lot about operands and CLDR plural rules syntax will help you recognize how much effort CLDR puts into that.

And ECMA402 Intl.PluralRules correctly handles formatted numbers, so minimumFractionDigits will impact your result:

(new Intl.PluralRules("en", {minimumFractionDigits: 1}).select(1))
"other"
(new Intl.PluralRules("en", {minimumFractionDigits: 0}).select(1))
"one" 

[zbraniecki]

Does CLDR even distinguish this case? AFAICS it doesn't.

It does. You can read more about it in https://www.unicode.org/reports/tr35/tr35-numbers.html#Language_Plural_Rules

In particular, a lot about operands and CLDR plural rules syntax will help you recognize how much effort CLDR puts into that.

And ECMA402 Intl.PluralRules correctly handles formatted numbers, so minimumFractionDigits will impact your result:

(new Intl.PluralRules("en", {minimumFractionDigits: 1}).select(1))
"other"
(new Intl.PluralRules("en", {minimumFractionDigits: 0}).select(1))
"one" 

[spookylukey]

@zbraniecki - thanks so much for that, it looks like I was misunderstanding based on what I was reading from Python Babel's implementation. In fact on further investigation Babel does also seem to support this correctly if you use Decimal rather than float.

[spookylukey]

@zbraniecki - thanks so much for that, it looks like I was misunderstanding based on what I was reading from Python Babel's implementation. In fact on further investigation Babel does also seem to support this correctly if you use Decimal rather than float.