Relationship between main pointer event and coalesced events

[jakearchibald]

It isn't clear to me how the coalesced event list works.

If I have a trusted event from pointermove, should event.getCoalescedEvents() ever return an empty array? It does on Firefox Android.

Let's say event.getCoalescedEvents() returns an array of two items:

const [one, two] = event.getCoalescedEvents();

What's the relationship between the single x/y points expressed by event, one and two? Is either one or two representing the same x/y point as event? If not, where does the event x/y point appear in the sequence?

[jakearchibald]

It seems to me that event.getCoalescedEvents() should return a complete set of x/y points for that event, meaning that the last x/y point returned by event.getCoalescedEvents() will be the same x/y point expressed by event itself.

This means that event.getCoalescedEvents() shouldn't return an empty array, as it would always at least contain the x/y point expressed by event.

[patrickhlauke]

my understanding so far was that the array contains all points except the one that was actually dispatched as a "real" pointer event - i.e. coalesced list includes all "dropped" points that ended up being coalesced/quietly ignored leading up to the actual one that was sent.

as for Firefox/Android, I believe the implementation of coalesced and predicted events lists there is only a stub at the moment (but @smaug---- may know more). indeed, testing using https://patrickhlauke.github.io/touch/coalesced-predicted-events/ shows me no entries in the coalesced events lists when trying on Firefox/Android (though Firefox/Windows it works nicely, though still no predicted events there either).

[jakearchibald]

my understanding so far was that the array contains all points except the one that was actually dispatched as a "real" pointer event - i.e. coalesced list includes all "dropped" points that ended up being coalesced/quietly ignored leading up to the actual one that was sent.

So the order is [one, two, event]? That isn't clear to me from the spec, and I'm not sure it's developer-friendly either.

For example, if you want to do something with all the points, which seems like the common case, you end up doing something like this:

ctx.beginPath();
ctx.moveTo(previousEvent.clientX, previousEvent.clientY);
for (const event of [...mainEvent.getCoalescedEvents(), mainEvent]) {
  ctx.lineTo(event.clientX, event.clientY);
}
ctx.stroke();

Rather than:

ctx.beginPath();
ctx.moveTo(previousEvent.clientX, previousEvent.clientY);
for (const event of mainEvent.getCoalescedEvents()) {
  ctx.lineTo(event.clientX, event.clientY);
}
ctx.stroke();

It also doesn't fit right with the naming for me (I know I know, naming is hard), but you're getting "coalesced" events, not "skipped" events. If I'm asking for the things that were "coalesced" together to form event, it feels like it should include the final value too, right?

[jakearchibald]

https://static-misc-3.glitch.me/pointer-test/4.html - this logs true to the console if the last coleasced event has the same client x/y as the main event, and it seems to be true all the time on desktop and mobile.

I agree there's nowhere in the spec that say that's supposed to happy though.

[patrickhlauke]

so it's just me not understanding the finer points of our spec, which is not unusual. does point to the fact that this should be explicitly mentioned if that's indeed expected/wanted behaviour. trying to work out if this is what the old coalescing touches behaviour in iOS (e.g. https://dzone.com/articles/advanced-touch-handling-in-ios9-coalescing-and-pre-1) does. if so, makes sense to explicitly say in the definitions that the last item in the array matches the actual "real" event then. @mustaqahmed @flackr @smaug---- thoughts?

[jakearchibald]

so it's just me not understanding the finer points of our spec, which is not unusual.

I'm not criticising you at all! This should have been written down properly by Chrome folks. Unless I'm missing something, it's just not there in the spec, so it's totally understandable that Chrome and Firefox have different behaviours here.

[flackr]

I agree that the final event position itself should be in the coalesced events, note that this is not exactly the same as the final event itself as the coalesced event does have some fields which represent the entire coalesced event list. The drawing example in the spec assumes this.

[jakearchibald]

I guess we need some normative text that shows how coalesced event list is populated, that makes it clear that it'll never be empty for trusted events.

[NavidZ]

Let me quickly chime in here. The event that is fired by the browser right before rAF callback is not necessary the position of the real event. That is the event that the brower believes to be the right repesentative of the input stream at rAF timestamp. As a result browsers are free to do resampling of the coordinates and other attribtues as they wish. Coaleseced events (including the last event in the coalesced event list) are guaranteed to be the actual events received by the browser with no manipulation.

Since most browsers don't do re-sampling based on rAF today, the effect you see most often is that the last event in the coalesced event list matches to the main event in some attributes (such as coordiantes) but that is not guaranteed and shouldn't be what devs expect. Also note that even if a user agent does no resampling of the fired event it still doesn't mean that the last event in the coalesced event list exactly matches the event itself. As their movementX,Y will certainly be different if there is more than one event in coalesced event list.

In general as we tried to mention in the spec, devs should either look at the coaelsced event stream (and ignore the event itself) or work only with the event itself. Mix and matching these will not yield in any good outcome.

[jakearchibald]

That seems reasonable. Just to confirm: event.getCoalescedEvents() shouldn't return an empty array if the event was created by the browser as part of a pointermove or pointerrawupdate event. Is that true?

[mustaqahmed]

I think current spec wording doesn't require it. So if no event coalescing took place before firing a pointermove, either an empty list or a list with a single event (which represents the same fired event) can be expected.

The spec can add a requirement that the list will always be empty or non-empty in this case, but I am not sure if this is necessary.

[NavidZ]

I don't know what @jakearchibald meant by "the event was created by the browser".

But if we were to stick with the idealogy of devs should either work purely with the dispatched events alone or with the event.getCoaelscedEvents() then an event should always have a coalesced event. Otherwise this logic of if event.getCoaelscedEvents() is empty then use event will lurk into all codes.
I have a vague memory that it was called out somewhere in the spec but I don't seem to find it now as @mustaqahmed mentioned. Maybe it even never was there and it was always in my head. But this approach is visible in the example I added back then in the spec that mentions the coalesced events count.

[flackr]

But this approach is visible in the example I added back then in the spec that mentions the coalesced events count.

AFAICT this example only checks for the existence of the getCoalescedEvents method, not whether there were coalesced events.

[NavidZ]

AFAICT this example only checks for the existence of the getCoalescedEvents method, not whether there were coalesced events.

I'm referring to w/ one coalesced event for the pointerrawupdate events. Even though there is only one real event happening since the last dispatch it still fills out coalesced event list with one event which is not what @jakearchibald reported at the beginning for Firefox on Android where the coalesced events array of pointermove is empty sometimes.

[jakearchibald]

@mustaqahmed

either an empty list or a list with a single event (which represents the same fired event) can be expected.

"Either"? That seems bad for developers and interoperability. It should be spec'd to be one or the other, or it should be clear when it's one or the other.

[jakearchibald]

@NavidZ

I don't know what @jakearchibald meant by "the event was created by the browser".

Developers can call new PointerEvent(…) and pass in their own values. I'm talking about trusted PointerEvents created by the browser in response to a pointer move.

But if we were to stick with the idealogy of devs should either work purely with the dispatched events alone or with the event.getCoaelscedEvents() then an event should always have a coalesced event.

Agreed.

Otherwise this logic of if event.getCoaelscedEvents() is empty then use event will lurk into all codes.

That's what I had to add GoogleChromeLabs/pointer-tracker@434a8ec#diff-d5a4b3a0309f2337144861084c1014523cced561eb722cb9684c63f5e41e0567R52.

I think the spec needs an algorithm for "reacting to a pointer update", which details how values are updated and events are constructed and dispatched. This would include updating the coalesced event list, flushing deferred events, hit testing, targeting etc etc.

[flackr]

I'm not sure if I'd go as far as it having an algorithm it has to follow, but I agree that developers should be able to work solely off of the coalesced events if they wish to work with coalesced events. The following note in the spec suggests that the top-level event is a unified summary of the coalesced events:

The PointerEvent's attributes will be initalized in a way that best represents the events in the coalesced events list.

In particular, this note is there to suggest that the movementX / movementY should reflect the total movementX / movementY of the coalesced events. However, it seems like this won't be strictly guaranteed given the point @NavidZ made that the top-level event could be resampled to the frame time.

Considering resampling, I think we should still expect to see one real "raw" event in the coalesced events for each top-level move event. Perhaps we could say that the coalesced events should include all of the raw events which were included in the top-level event, even if none were coalesced?

[mustaqahmed]

Adding a normative text requiring "at least one event in the coalesced event list" sounds good.

One clarification though: this would apply only to pointermove and pointerrawupdate events, right? I mean for all other event types that are never coalesced, an empty list still looks cleaner to me.

[jakearchibald]

@flackr

I'm not sure if I'd go as far as it having an algorithm it has to follow

I think it's needed. There's so much behaviour that isn't written down, and I think that might be causing a lot of the interop issues around pointer events.

For instance, the spec says that pointermove can be delayed, but it doesn't give the conditions of that delay. Can two pointermove events be given a different delay that results in them being delivered in a different order?

Can the pointermove in reaction to a button-down change happen after the related pointerdown event? Can it be delayed so much that it happens after a pointerup for that button?

What if the user presses a key on the keyboard and there's a pending pointermove? Does it flush that event?

There are browser behaviours and developer expectations here and as far as I can tell they're not in the spec. I guess you could answer them by sprinkling constraints throughout the spec, but I think it'd be better answered by an algorithm that details what should happen when the browser receives a change in pointer state. The algorithm could still allow for desirable differences in UAs, like delays, but with requirements that developers have come to rely on.

[patrickhlauke]

I think some of these questions may need to be addressed more upstream in UIEvents in general, not sure