Missing guidelines/patterns for locale-sensitive testing

[gibson042]

Nearly all ECMA-402 APIs (and some in ECMA-262) have implementation- and locale-dependent ("ILD") behavior, and those details change over time (e.g. with new browser customizations or CLDR/ICU releases). Relying on "golden output" is therefore brittle, but we also want to achieve good coverage.

There are options such as looking at stable substrings or other parts of output (such as month names), comparative testing across related inputs (e.g. that each setting for an input option produces a distinct output), general metamorphic testing, and broader decisions to make such as the set of locales and locale options to utilize (and whether or not to always utilize all of them).

Guidance should be added to CONTRIBUTING.md in the form of both explanations and examples. A proposed first step in establishing such guidance is to start with a relatively simple ILD API, and Intl.RelativeTimeFormat seems like a good candidate because it supports a small number of options (just numberingSystem, style "long" vs. "short" vs. "narrow", and numeric "always" vs. "auto") and its formatted output includes both numeric and textual parts (the former of which depends upon and can therefore be compared against Intl.NumberFormat functionality). Let's take a look at the existing tests and see what conclusions can be drawn from them.

[littledan]

How does shifting the tests to wpt help?

[ptomato]

I guess the context for this question was my comment in this WPT pull request? I don't think we need to shift tests to WPT as such, but since we are trying to find new strategies to test ECMA-402 without relying on golden output, I'd say that if people specifically want tests that rely on golden output, test262 is not the right place for that — because those tests do not accurately measure conformance with ECMA-262 or ECMA-402. Stability in ILD strings seems like it's mainly important to engines that run in browsers, so WPT seems like it could be the right place, but that's ultimately up to the people with merge privileges in WPT.

[annevk]

How is it not relevant to other engines?

[ptomato]

One place where compatibility issues arise with ILD string changes is from serializing and rehydrating web pages. As an example of an engine where that isn't a concern: I'm involved with the embedded SpiderMonkey in the GNOME desktop, for example. We are not serializing web pages there, but just displaying the ILD strings directly to users, in which case fixes that make the ILD output more consistent with what users in that locale expect are usually welcome.

[annevk]

Why could the same kind of thing not happen? Say you get a string and split it on U+0020 but then later on the string no longer contains that code point and your app is broken. Perhaps if you take extreme care to never operate on the returned strings you're good, but that seems unlikely to be generally true. And that concern definitely applies well beyond browsers.

[ptomato]

Yes, the same thing could happen, but when you re-parse an ILD string in an app I think you're probably more likely to be aware you're doing something iffy. Anyway, I said "mainly important to browsers" not "exclusively important" 😄