Extend Formatters to allow for token formatting

[zbraniecki]

UPDATE:

• Proposal
• Latest slides - Nov 2015

---

DateTimeFormat and NumberFormat are design to provide an opaque string as an output that is not meant to be manipulated by the consumer.
Unfortunately, that is impossible if one needs to format the output and that currently forces very dirty solutions that may break internationalization.

Example use case, is formatting minutes token to be smaller or different color than hours in hour:minute string.

var formatter = Intl.DateTimeFormat(navigator.languages, {
  hour: 'numeric',
  minute: 'numeric'
});
var string = formatter.format(new Date());
element.innerHTML = string; // we want to display hour token with bold font.

Modern user interfaces do that all the time - examples are Material Design in Android, modern Windows Mobile UI and PS4.

My proposal is to allow for token format strings to be available as option for the formatter. It could look like this:

var formatter = Intl.DateTimeFormat(navigator.languages, {
  hour: 'numeric',
  minute: 'numeric',
  format: {
    hour: '<strong>$&</strong>' // similar to String.replace(hour, '<strong>$&</strong>')
  }
});
var string = formatter.format(new Date()); // return '<strong>14</strong>:12' in pl

For non HTML outputs, the formatting may involve console formatting characters for terminal UI apps.

[caridy]

I'm not sure about this at first glance, but it seems that we have some similarities between what you describe here and what a tagged template string does (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/template_strings#Tagged_template_strings), which ultimately give you more control over the formatting of the final output. Maybe we should think in those terms instead of introducing yet another template form.

Aside from that, having the custom format in the constructor prevent you from reusing the formatter in multiple places if the custom format is different.

[zbraniecki]

I agree, I didn't spend too much time thinking about how the API should look like, and I'd be happy to reuse template strings as long as it allows for token formatting in datetime string construction.

You're also correct about format in constructor, but I would say that it may be a good tradeoff. Formatters should be consistent and if a customer needs to produce two different strings according to different formattings, he may need two different formatters.
But if it's easier to add a format template string as an argument for formatter.format, I'm totally fine with that.

[srl295]

ICU (and Java) use a FieldPosition construct. perhaps something like this:

var formatter = Intl.DateTimeFormat(navigator.languages, {
  hour: 'numeric',
  minute: 'numeric'
});
var fieldPosition = {};
var string = formatter.format(new Date(), fieldPosition);
// string = "12:34"
// fieldPosition = { hour: [ 0,1], minute: [4,5] };

Not sure what the right idiom is, but that's conceptually the data you might want. THen you can do whatever with the result.

[zbraniecki]

Awesome!
The API feels a bit awkward for JS, but it does solve the problem.

@caridy - do you have any preference on how the API should look like? I definitely like the formatting to be on formatter.format, but I'm not sure if it's better to have the second argument alter the returned string or stay close to ICU/Java implementation and provide additional information that allow the user to modify the string later.

[caridy]

let me ask around tomorrow, and see if someone has a good idea about the API. I'm ok with a second argument, but I'm not ok mutating that argument as proposed above.

[zbraniecki]

Ok, so would you prefer the approach proposed by @srl295 - we return the start and end indexes of the tokens?

[zbraniecki]

@srl295 , @caridy : I wrote a patch proposal for the spec to add fieldPosition to FormatDateTime function.
https://github.com/tc39/ecma402/compare/master...zbraniecki:dtformattokens?expand=1

Can you tell me how does it look at what's the next step?

I need to get something in our platform soon and would like to make it in line with the committee thinking about this feature.

[zbraniecki]

And here is the patch for Intl.js polyfill - https://github.com/andyearnshaw/Intl.js/compare/master...zbraniecki:dtformattokens?expand=1

[caridy]

First thing first, we need to agree on the API first. I know I dropped the ball on this one since it wasn't at the top of my list, but I can certainly spend some time on this api this week. Please, add a comment here with the exact change (minimum amount of text) so we can discuss it, reading the spec diff :) to understand the proposal is not optimal.

[zbraniecki]

Sure!

The proposal is to add an optional, second argument for Intl.DateTimeFormat.format(date, (bool) fieldPosition). If added, the return value is an object with two fields - string and fieldPosition where fieldPosition is an object matching ICU FieldPosition type [0].

Example:

var f = Intl.DateTimeFormat('ar', {
  hour: 'numeric',
  minute: 'numeric'
  hour12: true
});

var res = f.format(date, true);
// { string: '٤:١٩ م',
//  fieldPosition:
//   { hour: { beginIndex: 0, endIndex: 1 },
//     minute: { beginIndex: 2, endIndex: 4 },
//     dayperiod: { beginIndex: 5, endIndex: 6 } } }

I considered using a separate function, but felt like it's easier to extend what FormatDateTime returns.
Let me know what you think.

[0] http://icu-project.org/apiref/icu4c/classicu_1_1DateFormat.html#a620a647dcf9ea97d7383ee1efaf182d1

[zbraniecki]

Oh, and the reason I went for beginIndex and endIndex is because that's what ICU returns. I'm totally ok converting it to an array like in @srl295 proposal.

[caridy]

@zbraniecki few notes:

• fieldPosition structure seems fine.
• string is probably no good, maybe value, or result, or just the toString() version of the object?
• boolean arguments are very very ambiguous, and we have been trying to escape that trap for all new APIs.
• maybe a separate method instead of reusing format() because that one is focused on producing a string representation of the date value.

/cc @ericf @rwaldron can you guys chime in?

[zbraniecki]

Yeah, as I'm playing with it, I am changing my mind, how about:

var f = Intl.DateTimeFormat('ar', {
  hour: 'numeric',
  minute: 'numeric'
  hour12: true
});

f.format(date); //  ٤:١٩ م

var res = f.formatWithPosition(date);
// { value: '٤:١٩ م',
//  fieldPosition:
//   { hour: { beginIndex: 0, endIndex: 1 },
//     minute: { beginIndex: 2, endIndex: 4 },
//     dayperiod: { beginIndex: 5, endIndex: 6 } } }

Alternatively we could do format and pattern methods, but that would require user to launch the same code twice - once to format, and second time to get the pattern (I don't think retrieving a pattern without retrieving a formatted string is going to be useful), so formatWithPosition sounds more applicable.

[zbraniecki]

Here's Intl.js polyfill branch with the latest proposal: https://github.com/zbraniecki/Intl.js/tree/dtformattokens

[zbraniecki]

For C++, I'm using udat_formatForFields [0] and it returns more field types than we recognize in Intl[1].

My suggestion is to map it to field names this way:

{
  0: 'era',
  1: 'year',
  2: 'month',
  3: 'day',
  4: 'hour',
  5: 'hour',
  6: 'minute',
  7: 'second',
  9: 'weekday',
 14: 'dayperiod',
 15: 'hour',
 16: 'hour',
 17: 'timeZoneName',
}

and ignore the rest for now.
In the future we may want to add things timeSeparator and millisecond.

[0] http://www.icu-project.org/apiref/icu4c/udat_8h.html#a4bc9d9661c115dcb337803bc89730b3a
[1] http://bugs.icu-project.org/trac/browser/icu/trunk/source/i18n/unicode/udat.h#L488

[zbraniecki]

@caridy - what do you think about formatWithPosition?

[zbraniecki]

Here's the proposal for ecma402 spec patch to add this: https://github.com/zbraniecki/ecma402/tree/dtwithposition

Opinions? @caridy , @rwaldron , @srl295 , @domenic, @rxaviers ?

It's kind of top priority for me because without that our platform has very hacky solution to style tokens in time strings and we do this in multiple places. The result is that in some locales our UI is broken and I'd like to get this feature landed for the next release.

[ericf]

@zbraniecki, @caridy and I spent some time discussing this today. The first thing is we both agree that the ability to do "rich-text" formatting is an important feature. I had the same initial thought as @caridy about how this feels similar to a template literal tag function… but after going further and prototyping how a library like react-intl would use this we landed on something slightly different from your proposal.

Instead of having a fully formatted string with a collection of positions, we ended up wanting an array for formatted part descriptors.

let dateFormat = new Intl.DateTimeFormat('en');
let date = Date.now();

let formattedDate = dateFormat.formatToParts(date)
    .map(({value}) => value)
    .reduce((string, part) => string + part);

let fancyFormattedDate = dateFormat.formatToParts(date)
    .map(({type, value}) => {
        switch (type) {
            case 'month': return `<b>${value}</b>`;
            default     : return value;
        }
    })
    .reduce((string, part) => string + part);

console.log(dateFormat.format(date)); // "10/30/2015"
console.log(formattedDate);           // "10/30/2015"
console.log(fancyFormattedDate);      // "<b>10</b>/30/2015"

The bikeshed hasn't been painted here yet, but we think having an array of formatted part descriptors like this makes it easier to process each part and put them back together in a final string than trying to use substrings on positions. The parts can be processed left-to-right without worrying about throwing off the position indexes of the following parts when modifying the current part; e.g., wrapping it with HTML.

For the example above, the array of part descriptors might look something like this:

[
    {
        type: 'month',
        value: '10',
    },
    {
        type: 'separator',
        value: '/'
    },
    {
        type: 'day',
        value: '30',
    },
    {
        type: 'separator',
        value: '/'
    },
    {
        type: 'year',
        value: '2015',
    }
]

Note how the values are always strings. type is probably not the correct label, but this is mainly to illustrate this idea compared with the positions idea.

[caridy]

To add to that, we think formatToParts() (we can bikeshed on that method name) will be a method of DateTimeFormat, NumberFormat, RelativeTimeFormat, etc. In general, if you can format something, you should also be able to format the parts. Internally, format() and formatToParts() relies on the same abstract operation, but format() does put the parts together using the algo described above for formattedDate.

[zbraniecki]

Awesome! Thanks a lot for this brainstorming. I love the API and I'll be happy to update my proposal to it.

One question about naming of type

That's unfortunately a little bit more complex to implement and I'm wondering how you'd like to approach.
We generally have two sources of datetime formatting patterns:

• JS pattern string (like: "hmsv": "h:mm:ss a v" or "GyMMMMEd": "E, d MMMM y G")
• ICU's udat_formatForFields ( [0] )

Both approaches make it easy to index positions of date time components, but not much for the separators.

While ICU returns "timeSeparator" field [0], everything else is a fair game. It's hard to know if the separator between day and month will be /, ., , or something else.

Because of that, would you just want to use a generic term separator for everything that is not a token?
It's a bit pity, because if someone would like to style time separator in 12<span>:</span>36 they'd have to find hour and minute token and the separator type between them (which may also not be present?).
But I don't really see a way to name all separators and their combinations (minute/dayperiod separator? weekday/year separator?)

[0] http://www.icu-project.org/apiref/icu4c/udat_8h.html#a4bc9d9661c115dcb337803bc89730b3a
[1] http://www.icu-project.org/apiref/icu4c/udat_8h.html#adb09b47d4576513229f83f2e8f507fc2a948cc45ccae55dcf5c094e3c33a2c3d7

[caridy]

I'm not sure I follow your question, but I can tell you that separator is probably not a good type, maybe just text, or token, or something is better to signal that it is part of the formatted value that doesn't correspond to any of the units used to produce the output.

[ericf]

@caridy I think the concept of separator seems correct. There might more types like this as well which cover the concept of "text".

@zbraniecki Are you thinking we'd need something like this?

{
    type: 'field',
    field: 'month',
    value: '10'
}

[zbraniecki]

No, what I meant is that I'm not sure if "Wednesday, 03/2015 12:32 am" should be reported as:

[
  { type: 'weekday', value: 'Wednesday' },
  { type: 'separator', value: ', ' },
  { type: 'month', value: '03' },
  { type: 'separator', value: '/' },
  { type: 'year', value: '2015' },
  { type: 'separator', value: ' ' },
  { type: 'hour', value: '12' },
  { type: 'separator', value: ':' },
  { type: 'hour', value: '32' },
  { type: 'separator', value: ' ' },
  { type: 'dayperiod', value: 'am' },
]

or should we try to name separators like timeSeparator, dateSeparator (ICU has a special token named timeSeparator). But I guess it doesn't make much sense since there may be all combinations of separators.

So the question @caridy asked stays - type separator or type text or sth else?

[zbraniecki]

I got Intl.js polyfill to support the proposed formatToParts. Can you look at it and provide feedback? I'll write a patch to spec next.

[ericf]

@zbraniecki I'm wondering how LRM (\u200E) and RLM (\u200F) characters should be handled when we format to parts. Do you have any thoughts on this and whether the marker chars should be "sticky" to the field types, or separator types?

I'm not an expert how these bidi markers actually work and what happens when we process the parts by wrapping them with a span. e.g.:

new Date(0).toLocaleString("ar",{month:"numeric",day:"numeric",year:"numeric"}).replace(/\u200F/g, '[RLM]');

// "٣١[RLM]/١٢[RLM]/١٩٦٩"

If we wanted to bold the month and create an HTML string, would the marker be part of the <b> element's children or the / separator?

This is related to: #28

[zbraniecki]

I believe that the snippets I provided are the closest thing we can do to locale-independant behavior. It may have exceptions (which may need if (lang in [])), but if you want to capture more than the bare parts, you may use such snippets.

My understanding is that the API provides the lowest common denominator and allows for higher level APIs to be built on top.

[zbraniecki]

Intl.DateTimeFormat.prototype.formatToParts has been exposed to the Web in today's Firefox - http://hg.mozilla.org/mozilla-central/rev/3f47a92541c8

[zbraniecki]

Intl.NumberFormat.prototype.formatToParts has landed in SpiderMonkey today -
https://hg.mozilla.org/integration/mozilla-inbound/rev/f849271896d3

[zbraniecki]

We're waiting for second vendor to step in now. @jungshik, @littledan - do you know if there's a chance to get this into V8 even in an experimental form so that Google would be comfortable recommending it for stage4?

The last call to get this into 4th edition is probably at January's TC39 meeting.

[littledan]

@zbraniecki Looks like the ICU part has its upstreaming still in progress. It'll be easier for us to implement in V8 after that's done. Is there any particular reason why it has to get to Stage 4 at this meeting? Could we wait for a V8 implementation until the ICU side work is more mature?

[zbraniecki]

Yeah, certainly.

I do think about ES releases as milestones, mostly because that's how the market treats them. While I'm not worried about V8 shipping features quickly, I believe that if we miss the train for ES2017 with formatToParts and PluralRules, it significantly increases the chance that it'll add a whole year until all modern JS engines will ship it and that we won't be able to rely on it until at least Q3 2018.

My hopes were for those two features to be in a good shape for stage 4 in time for ES2017. If you believe that it's not worth it and if you don't share my worry about market adoption, I trust your experience on the matter.

[littledan]

I think you've been doing a good job building towards getting browsers to support this by making sure there's an implementation in Mozilla and getting the ICU part upstreamed properly. I haven't seen a lot of evidence that browsers are waiting for the annual cutoff to implement features, on the other hand. I am not sure how long it will take for people to be able to depend on the feature--IE11 will probably never support it, but that browser may be in use for some time.

[jungshik]

Note that ICU is NOT a blocker for v8 because v8 does not plan to use ICU's C API. The ICU change made by Mozilla is just to port an existing C++ API to C API. What I'm not yet 100% sure is the practical usefulness of NumberFormt.formatToParts (and I haven't fully digested the proposed API, yet)

[zbraniecki]

Hi all,

@littledan pointed out that we failed to document the rationale for the decision to add NumberFormat.prototype.formatToParts alongside with the DateTimeFormat's one.

We did talk about it quite a bit during TC39 meetings, and showed examples as we were requesting the stage advancements, but never wrote down the rationale, so here's my attempt to capture the spirit of it for the record:

1. Rich formatting for numbers using map/reduce model

At the beginning, in September 2015, we discussed several approaches to allow for internationalization-friendly rich formatting of the data formatted by the Intl formatters.

It was fairly clear to all of us (@ericf, @caridy, @stas, @srl295 and me) that we're aiming at the most simple, chainable, solution that will allow users to achieve the simple, most common, tasks easily, while allowing more complicated ones if needed (with accompanying complexity).

The feature request was based on my experience of working on Firefox OS where we encountered a number of UX/design requirements for rich formatting including:

- Dates (ex. "May <strong>05</strong> 2014")
- Time (ex. "09:49 <i>pm</i>")
- DateTimes (ex. "<strong>Monday</strong>, 09/12/2015")
- Numbers (ex. "1 000.<strong>35</strong>")
- Currencies (ex. "<strong>EUR</strong> 35")
- Percent (ex. "35.4 <strong>%</strong>")

The initial focus, due to requirements of my use case project, were primarily focused on DateTimeFormat API, but alongside, @stasm wrote the spec proposal for the corresponding NumberFormat API.

On October 30th 2015, @ericf came up with an API that serves the need elegantly in our opinion allowing for any call to format function to be replaced with a chained formatToParts/map/reduce sequence.

2. Cohesive API

Although secondary to the first reason, I believe it's worth bringing up the principle of least astonishment.
formatToParts is often requested for formatting dates and units, and @caridy made the decision to provide formatToParts to all formatters.
I believe that there is a value in covering every Intl API that has format function with a formatToParts counterpart.

Known limitation:

The formatToParts/map/reduce chain for NumberFormat comes up with a particular challenge that other APIs either existing (DateTimeFormat) or planned (ListFormat, UnitFormat, RelativeTimeFormat, DurationFormat) do not share.
In particular, it formats a full number that has components. For example "1 000 000.00" can be formatted on two levels of granularity - into ["1", "000", "000", ".", "00"] and ["1 000 000", ".", "00"].
Each one allows makes different thing easy. The former makes it easy to style each component of the number, the latter makes it easy to format the whole integer part of the number with one token.

The difference is that the former tokenization allows to tokenize the whole integer, although in a bit awkward way, while the latter does not allow for the styling of each component.

ICU provides an API that allows to workaround this challenge by producing overlapping list of tokens where you can get start/end positions for both, the whole integer and its components.
The challenge with it, is that it cannot be used to map/reduce the list into a string.

In the end, we ( @srl295 , @caridy , @ericf , @stasm and me) came to conclusion, that while such API may be useful, it's a separate API and should not be using formatToParts name.
We see value in getting formatToParts for both of the reasons above into the spec, and if there's interest in getting an overlapping API just for NumberFormat due to its unique use cases, we should explore it separately.

I hope it helps!

[zbraniecki]

Intl.NumberFormat.prototype.formatToParts is now shipped in two engines - SpiderMonkey and V8 - behind the flag.
I would like us to request Stage 4 at the next TC39 meeting.

[rxaviers]

I took the liberty to update this issue description and include the proposal link.

[caridy]

@zbraniecki can you add the agenda item for this? If you don't plan to be in the next meeting (Boston) I can do the update.

Also, we need to work on the spec text, to get the PR ready before the next meeting, many things has changed since the revert, and we need to work things out.

[zbraniecki]

@stasm would you be interested in updating your PR to the current master?

[stasm]

I filed #160. I had to refresh my memory on what happened to the original PR a year ago. If the summary in #160 is off, please correct me.

[zbraniecki]

Intl.NumberFormat.prototype.formatToParts reached Stage 4 at today's TC39 meeting. I believe once we merge the PR, we can close this issue! :)

[caridy]

two years, yay!

merged fe7b0a4

[zbraniecki]

With https://hg.mozilla.org/integration/autoland/rev/353baf72f789 landed, SpiderMonkey now exposes both without any flag.