Introduce pointerrawupdate to address the high frequency usecases #260

NavidZ · 2018-07-04T16:45:07Z

Fixes #214

NavidZ · 2018-07-10T18:27:12Z

@smaug---- what do you think of pointerrawmove to address the low latency usecases? I tried to remove the overhead of hittesting on every one of those events by restricting it to only capturing mode.
I wanted to start the implementation in Chrome and wanted to know what you think of it first.

smaug---- · 2018-07-10T22:50:19Z

(discussion about features shouldn't happen in PRs, but in issues)

smaug---- · 2018-07-24T15:26:14Z

extension.html

+           <code>pointerrawmove</code> will get fired as soon as an attribute is changed
+           and as frequent as the javascript can handle the events.
+           Because of that the target of <code>pointerrawmove</code> events might be different from the <code>pointermove</code> events
+           due to the fact that <code>pointermove</code> events are being sent once every frame and the final position of the even


pointermove isn't defined to be sent once every frame.

You are right. This part is not yet specced. Not sure how to get around it. More on it in the other comment.

smaug---- · 2018-07-24T15:28:37Z

extension.html

+           Because of that the target of <code>pointerrawmove</code> events might be different from the <code>pointermove</code> events
+           due to the fact that <code>pointermove</code> events are being sent once every frame and the final position of the even
+           which is used for finding the target could be different from its coalesced events.
+           Note that if the javascript handlers are slow, <code>pointerrawmove</code> events can also get coalesced.


ok, this makes the behavior from specification rather odd. pointermove isn't defined to be only once per frame and then pointerrawmove may get coalesced too, so in practice the events could be exactly the same.

The idea was that pointermove will get delayed at the very least until right before rAF. But pointerrawmove will not have that delay at all. However both of these events might get unlimited delay and get coalesced during that time based on the how much the javascript code keeps the code busy. Does this at least make the intention for this event clear and how it is different from pointermove?

Now regarding how to spec it, the problem is that rAF related stuff is not fully specced yet and I believe I need to change HTML spec for this and reference https://html.spec.whatwg.org/#event-loop-processing-model here.

I understand what you try to spec, it is just a tad hard to spec when pointermove isn't spec'ed to work exactly how it works in Blink (nor in Gecko). Pointerevents spec says "A user agent MUST fire a pointer event named pointermove when a pointer changes coordinates." That actually kind of prevents firing only at rAF time. So I guess we should fix that too.

I tried to explain it better and also added a new pointermove description. Let me know what you think.

patrickhlauke · 2018-09-07T08:00:00Z

@smaug---- there's no rush on this as it targets the extension/possible v3 stuff, but...is this PR now looking better? can it be merged?

NavidZ · 2018-10-02T15:44:33Z

@smaug---- let me know if the latest of this change captures what you had in mind.

smaug---- · 2018-10-06T20:53:17Z

extension.html

+      <h2>Extensions to the Pointer Event types</h2>
+      <p>Below is the new event type that is added to the <a href="https://w3c.github.io/pointerevents/index.html#pointer-event-types">existing pointer event types</a>.</p>
+      <section>
+        <h3>The <dfn><code>pointermove</code> event</dfn></h3>


I don't understand the text. "Below is the new event type..." "the pointermove". pointermove sure isn't a new type.

smaug---- · 2018-10-06T20:54:15Z

extension.html

+        Additionally one <code>pointermove</code> MUST be fired when pointer changes coordinates, pressure, tangential pressure, tilt, twist, or
+        contact geometry (e.g. <code>width</code> and <code>height</code>) and the circumstances produce no other pointer events defined in this specification.
+        This <code>pointermove</code> that is not caused by button state change MUST be fired only once right before firing an event caused by button state change or
+        <a href="https://html.spec.whatwg.org/#event-loop-processing-model">animation frame callbacks</a>. If there are mutliple changes to these attributes


smaug---- · 2018-10-06T20:57:38Z

extension.html

+        <p>A user agent MUST <a href="https://w3c.github.io/pointerevents/index.html#firing-events-using-the-pointerevent-interface">fire a pointer event</a>
+           named <code>pointerrawmove</code> when a pointing device attribute
+           (i.e. button state, coordinates, pressure, tangential pressure, tilt, twist, or contact geometry) is changed.
+           As oppose to <code>pointermove</code> which could be delayed until an animation frame for changes in these attributes,


this talks about "could", yet the text above uses "MUST be fired only once"

smaug---- · 2018-10-06T20:59:02Z

extension.html

+           and as frequent as the javascript can handle the events.
+           Because of that the target of <code>pointerrawmove</code> events might be different from the <code>pointermove</code> events
+           due to the fact that <code>pointermove</code> events are being coalesced and the final position of the event
+           which is used for finding the target could be different from its coalesced events.


It would be good to explain how coalesced events are pointerrawmove differ. For example, other than .target are the properties the same?

smaug---- · 2018-10-06T20:59:54Z

extension.html

@@ -90,7 +121,7 @@ <h2>Extensions to the <code>PointerEvent</code> interface</h2>
 The dispatched event's attributes will be initalized in a way that is best representative of all the coalesced events.
 For example its <a href="https://www.w3.org/TR/pointerlock/#widl-MouseEvent-movementX">movementX</a> and <a href="https://www.w3.org/TR/pointerlock/#widl-MouseEvent-movementY">movementY</a> ([[!POINTERLOCK]]) COULD be the sum of those of all the coalesced events.
 None of the events in the sequence will have (nested) coalesced events, so <code>getCoalescedEvents</code> returns an empty sequence for them.
-This API always returns at least one coalesced event for <code>pointermove</code> events and an empty list for other types of <code>PointerEvents</code>.</p>
+This API always returns at least one coalesced event for <code>pointermove</code> and <code>pointerrawmove</code> events and an empty list for other types of <code>PointerEvents</code>.</p>


Why we need this for pointerrawmove?

I guess the same question would also hold for a pointermove that only has one coalesced event. Because in that case pointermove itself is that coalesced events. I believe the answer is that both pointerrawmove and pointermove may get coalesced. So in javascript code it is easier to always handle coalesced events of these events (if they care about that precision) instead of a check for the size of the coalesced events and if it is 0 then use the actual event itself. That is the only reason I can think of. What do you think? Is this way more ergonomic and easy to use (like the sample drawing code later in this doc)?

smaug---- · 2018-10-06T21:00:41Z

extension.html

@@ -114,7 +145,7 @@ <h2>Extensions to the <code>PointerEvent</code> interface</h2>

    <section>
      <h2>Timing of coalesced event firing</h2>
-      <p>Coalescing events only happens for <code>pointermove</code> events. All the coalesced events should be fired before the next frame or before any <code>pointerup</code>, <code>pointerdown</code>, <code>pointercancel</code> events. Note that user agents don't need to hit-test all the coalesced events as none of them will have null targets.</p>
+      <p>Coalescing events only happens for <code>pointermove</code> events. All the coalesced events should be fired before the next frame or before any <code>pointerup</code>, <code>pointerdown</code>, <code>pointercancel</code> events. Note that user agents don't need to hit-test all the coalesced events as all of them will have the same target as the dispatched event.</p>


... especially when this talks about coalescing happening for pointermove events only.

Fair. I forgot to update this. Particularly I already talked about the dispatch time of these coalesced events before. I removed that sentence from this paragraph.

NavidZ · 2019-01-02T17:25:18Z

@smaug---- could you take a look at this one and see whether I addressed your concerns?

smaug---- · 2019-01-03T13:09:32Z

extension.html

+           Because of that the <code>target</code> of <code>pointerrawmove</code> events might be different from the <code>pointermove</code> events
+           due to the fact that <code>pointermove</code> events are being coalesced and the final position of the event
+           which is used for finding the <code>target</code> could be different from its coalesced events.
+           Note that if the javascript handlers are slow, <code>pointerrawmove</code> events can also get coalesced.


"Note that if the javascript handlers are slow, pointerrawmove events can also get coalesced." what does this actually mean. What is slow JS? handler taking more than 16ms? handler forcing rAF to happen later? or what? And what is the reason for this coalescing? If the page wants raw events, it should get raw events. That would be simpler and easier to understand API.

Let's discussion a little more about this. Just to make sure whether we should change the design and logic around this event or just the text here.

I agree that when the page does express the desire to get the pointerrawmove events then presumably they have a fast handler that doesn't take long. But imagine a case that the app is not well written and takes say a few milliseconds to handle even a pointerrawmove and I have a 1000Hz mouse and I'm interacting with the page. So in this case user agent is sending 1000 move events to the page (assuming mouse is moving around) or 1 event every millisecond. So user agent either should drop these raw move events or queue them forever but the queue is only growing in that case (as the incoming events are faster than the handled rate). So user agent should either drop some of the rawmove events or coalesce them and expose those coalesced events with getCoalescedEvents APIs. right?

But if events are coalesced, when does that happen? When is it ok for UA to coalesce?
Is it ok to fire just one pointerrawmove per frame and coalesce the rest. Should the spec be a bit more precise. Something like... if UA detects that rendering will be updated and there are pending pointerrawmoves, fire just one event and coalesce the rest in the queue?

I see. So in general you are okay with coalescing the rawmove events. The only problem is a reasonable definition around it and when to do the coalescing. Let me take another stab at the text.

Sorry for the delay. I pushed a new commit. I used this sentence here instead: "Note that if UA detects the thread running javascript handlers for this event is not idle, pointerrawmove events will get coalesced and they will all be delivered as coalesced events of one pointerrawmove event as soon as the thread goes idle". Is that more precise?

Rather vague still, since "thread" isn't defined. If we want to define this properly, it should probably talk about tasks and task queues.
Also, if there are pending idleCallbacks, will those cause coalescing?

@smaug---- How about now?

smaug---- · 2019-03-13T21:49:43Z

extension.html

+        Additionally one <code>pointermove</code> MUST be fired when pointer changes coordinates, pressure, tangential pressure, tilt, twist, or
+        contact geometry (e.g. <code>width</code> and <code>height</code>) and the circumstances produce no other pointer events defined in this specification.
+        This <code>pointermove</code> that is not caused by button state change MUST be fired only once right before firing an event caused by button state change or
+        <a href="https://html.spec.whatwg.org/#event-loop-processing-model">animation frame callbacks</a>. If there are multiple changes to these attributes


This doesn't seem to follow what implementations do. If there is nothing animating, *move events still fire at some point.
And I think events should fire sooner even in case there is something animating. More like before resize steps.

What does this have to do with whether something being animated or not? Event if nothing is animating rAF is still there even if it doesn't run any callbacks in js. Can you explain what you tested? Chrome always does delay pointermoves with the current implementation to right before rAF.

smaug---- · 2019-03-13T21:50:44Z

extension.html

+        <p>A user agent MUST <a href="https://w3c.github.io/pointerevents/index.html#firing-events-using-the-pointerevent-interface">fire a pointer event</a>
+           named <code>pointerrawmove</code> when a pointing device attribute
+           (i.e. button state, coordinates, pressure, tangential pressure, tilt, twist, or contact geometry) is changed.
+           As oppose to <code>pointermove</code> which is delayed until an animation frame reached or button change events happen,


yeah, this doesn't seem to be what implementations do. (tested also Chrome)

Similar to the comment above. What is the scenario that you are testing that is not doing this?

smaug---- · 2019-04-01T19:27:30Z

extension.html

-        <a href="https://html.spec.whatwg.org/#event-loop-processing-model">animation frame callbacks</a>. If there are multiple changes to these attributes
-        the <code>pointermove</code> events before reaching animation frame callback or a button state change event,
-        these events will get coalesced and will be exposed via <a data-lt="getCoalescedEvents">getCoalescedEvents</a> API for the single dispatched <code>pointermove</code> event.
+        These events may be coalesced or aligned to <a href="https://html.spec.whatwg.org/#event-loop-processing-model">animation frame callbacks</a> basesd on UA decision.


smaug---- · 2019-04-01T19:27:44Z

extension.html

-        the <code>pointermove</code> events before reaching animation frame callback or a button state change event,
-        these events will get coalesced and will be exposed via <a data-lt="getCoalescedEvents">getCoalescedEvents</a> API for the single dispatched <code>pointermove</code> event.
+        These events may be coalesced or aligned to <a href="https://html.spec.whatwg.org/#event-loop-processing-model">animation frame callbacks</a> basesd on UA decision.
+        The coalesdced events information will be exposed via <a data-lt="getCoalescedEvents">getCoalescedEvents</a> API for the single dispatched <code>pointermove</code> event.


Introduce pointerrawmove to address the high frequency usecases

5060c4a

smaug---- mentioned this pull request Jul 10, 2018

Define a low latency event that isn't occur in the document lifecycle #214

Closed

Relax the capture requirements and remove bubbles=false

3f7e3f2

smaug---- requested changes Jul 24, 2018

View reviewed changes

Update pointermove description

5cf3b25

Add button state change to pointerrawmove as well

c04af8a

patrickhlauke added the v3 label Oct 2, 2018

smaug---- reviewed Oct 6, 2018

View reviewed changes

Address smaug's comments

3ac8302

smaug---- reviewed Jan 3, 2019

View reviewed changes

NavidZ added 2 commits January 24, 2019 09:16

Reword coalescing for pointerrawmove

9f056d4

Define coalesecing in terms of task queues

f886b2c

smaug---- reviewed Mar 13, 2019

View reviewed changes

Change the name of pointerrawmove to pointerrawupdate

2a71139

smaug---- reviewed Apr 1, 2019

View reviewed changes

NavidZ added 2 commits April 3, 2019 14:06

Fix typos and define ordering of pointerrawupdate and pointermove

7f23c30

Clarify the new event vs existing event types

82b34f7

smaug---- approved these changes Apr 8, 2019

View reviewed changes

NavidZ merged commit ded1e66 into w3c:gh-pages Apr 11, 2019

NavidZ deleted the pointerrawmove branch April 11, 2019 18:39

NavidZ changed the title ~~Introduce pointerrawmove to address the high frequency usecases~~ Introduce pointerrawupdate to address the high frequency usecases Apr 15, 2019

Roni-Carta mentioned this pull request Feb 18, 2022

Issue with "Event reference": Missing Event Handler pointerrawupdate mdn/content#13119

Closed

Introduce pointerrawupdate to address the high frequency usecases #260

Introduce pointerrawupdate to address the high frequency usecases #260

Conversation

NavidZ commented Jul 4, 2018

NavidZ commented Jul 10, 2018

smaug---- commented Jul 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhlauke commented Sep 7, 2018

NavidZ commented Oct 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NavidZ commented Jan 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment