Normative: Do not allow duplicate variants within the tlang component of a transformed_extensions

[jswalden]

That is, because en-emodeng-emodeng with duplicated emodeng variant is not structurally valid, neither ought de-t-en-emodeng-emodeng be valid. This fixes one of the issues raised in #330.

This patch's wording (and how it talks about restrictions on subtag sequences) is somewhat messy. Suggestions obviously appreciated for how to do this more clearly or concisely.

[leobalter]

Another one to bring up in the next meeting.

[FrankYFTang]

This line should be changed so reader won't misread it as there should be no variant tag at all.

[jswalden]

I keep staring at this and can't see this meaning in it...but also I wrote it. ^_^ I updated this with different wording that puts "duplicate" closer to the start; let met know if there's still a problem with that.

[jswalden]

@macchiati @zbraniecki Do either of you have comments on this structural-validity adjustment, and particularly how this intersects with BCP 47/UTS 35 validity requirements? The no-duplicates restriction at top level is already in specs; the one for tlang within a t extension is not yet specified, but it seems like a pretty natural fit if a top-level language tag is invalid for duplicate variants, it ought also be invalid inside a t extension's tlang.

[macchiati]

+ <li>the `unicode_language_id` within _locale_ does not contain ASCII case-insensitively equivalent `unicode_variant_subtag` subtags; and</li> Rather than repeat "ASCII case-insensitively equivalent" for the language around duplicates, I think it is cleaner to say in one place that validity tests are made as if the entire locale (or language) ID were lowercase. Thus *duplicate* as in "duplicate subtags" always means ASCII case-insensitively equivalent. Mark

…

On Thu, May 21, 2020 at 4:32 PM Jeff Walden ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In spec/locales-currencies-tz.html <#429 (comment)>: > </p> <ul> - <li>represents a well-formed "Unicode BCP 47 locale identifier" as specified in <a href="https://unicode.org/reports/tr35/#Unicode_locale_identifier">Unicode Technical Standard 35 section 3.2</a>,</li> - <li>does not include duplicate variant subtags, and</li> - <li>does not include duplicate singleton subtags.</li> + <li>_locale_ can be generated from the EBNF grammar for `unicode_locale_id` in <a href="https://unicode.org/reports/tr35/#Unicode_locale_identifier">Unicode Technical Standard #35 LDML § 3.2 Unicode Locale Identifier</a>;</li> + <li>_locale_ does not use any of the backwards compatibility syntax described in <a href="https://unicode.org/reports/tr35/#BCP_47_Conformance">Unicode Technical Standard #35 LDML § 3.3 BCP 47 Conformance</a>;</li> + <li>the `unicode_language_id` within _locale_ does not contain ASCII case-insensitively equivalent `unicode_variant_subtag` subtags; and</li> I keep staring at this and can't see this meaning in it...but also I wrote it. ^_^ I updated this with different wording that puts "duplicate" closer to the start; let met know if there's still a problem with that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#429 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCOG3LUVZFACUBGBADRSW2X3ANCNFSM4MQPV6RQ> .

[zbraniecki]

The no-duplicates restriction at top level is already in specs; the one for tlang within a t extension is not yet specified, but it seems like a pretty natural fit if a top-level language tag is invalid for duplicate variants, it ought also be invalid inside a t extension's tlang.

Is the de-duplication a canonicalization operation or validity check? I think in ICU4X we treat it as a canonicalization operation in accordance with the "Be liberal in what you accept, and conservative in what you send" principle.

I'd suggest doing that for both, language identifier and tlang along with case normalization and sorting.

[jswalden]

Is the de-duplication a canonicalization operation or validity check? I think in ICU4X we treat it as a canonicalization operation in accordance with the "Be liberal in what you accept, and conservative in what you send" principle.

It's a bad principle. :-) If you don't impose discipline in what you accept, senders will not self-discipline in what they send.

I'd suggest doing that for both, language identifier and tlang along with case normalization and sorting.

Duplicates in the language tag are excluded as a validity check -- and have been ever since BCP47. In principle that could be relaxed...but then you may need to drag along the rest of the ecosystem to also accept duplicates. That seems much harder than simply being more restrictive about what we accept, in the much-narrower area of tlang syntax. (Note that we currently are "conservative in what we accept" by excluding compatibility syntax, irregular tags BCP47 and ECMA-402 previously allowed, and extlang syntax, just off the top of my head.) To my knowledge, we have never loosened beyond what BCP47 describes, and coalescing duplicate variants in en-emodeng-Emodeng would be a loosening.

As usual @anba figured this out before both of us. :-)

[zbraniecki]

It's a bad principle. :-)

I don't agree.

That seems much harder than simply being more restrictive about what we accept, in the much-narrower area of tlang syntax.

Yes. that makes sense.

[sffc]

What is the status of this PR? Are we still waiting for feedback?

[jswalden]

I don't think we are any more -- the request for feedback was answered with "don't repeat the ASCII duplication language multiple times", the latest iteration states it only once, so this should be good to go, as far as I can tell.

[jswalden]

@sffc At this point, assuming you agree with my last comment, I think this can be merged. Unless adding test262 data at the same time is prerequisite for doing so, in which case I can file the issue and do the necessary work on it, or there's some other blocking reason I'm not aware of...

[sffc]

Is it correct to say that this is technically a normative PR but it reflects web reality and therefore is a no-op change from the implementation's point of view?

[jswalden]

@sffc It's a normative change. I don't know if it reflects web reality, but it seems doubtful to me there are users out there who knowingly pass duplicate variants in tlang. Chrome beta allows new Intl.Locale("de-t-en-emodeng-emodeng").toString() right now, as does Firefox. This would change that to instead throw RangeError.

[sffc]

Oh ok, so this is a normative PR. That means we need to do the dance of getting implementations, adding tests, MDN, etc. I opened tc39/test262#2857 for tests. If you're aware of implementation bug numbers, can you add them to the wiki? Thanks.

[Constellation]

FYI, JavaScriptCore rejects duplicate variants and throws RangeError from https://trac.webkit.org/changeset/266039/webkit

[jswalden]

SpiderMonkey bug for this is https://bugzilla.mozilla.org/show_bug.cgi?id=1670165 with patch posted. The test in it ought fairly readily be massaged into a test262 test -- I'll see about doing that, likely tomorrow.

[jswalden]

tc39/test262#2858 adds a test for this to test262.

[sffc]

ECMA-402 consensus: https://github.com/tc39/ecma402/blob/master/meetings/notes-2021-01-14.md#normative-do-not-allow-duplicate-variants-within-the-tlang-component-of-a-transformed_extensions-429

@jswalden to evaluate whether we need MDN. Tracking issue: tc39/ecma402-mdn#18

[sffc]

This achieved TC39 consensus today.