Pen Events API

[luisjuansp]

Saluton TAG!

I'm requesting a TAG review of Pen Events.

Some digital pens and pencils can not only provide input through their interaction with a digitizer but can also pair with a device so that additional signals can be received when a button is pressed. In some cases, the signal may relate to the pen or pencil but come from another source, e.g. a charger may send a signal that the pen or pencil has been docked or undocked. Native applications can use these signals to customize their behavior, but no corresponding events are available to web applications. Providing these new pen event primitives would enable web applications to achieve parity with native applications.

• Explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PenEvents/explainer.md
• Security and Privacy self-review²: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PenEvents/security-privacy.md
• GitHub repo (if you prefer feedback filed there): https://github.com/MicrosoftEdge/MSEdgeExplainers/issues
• Primary contacts (and their relationship to the specification):
  • Luis Juan Sanchez (@luisjuansp), Microsoft
  • Anupam Snigdha (@snianu), Microsoft
  • Bo Cupp(@BoCupp-Microsoft), Microsoft
  • Grisha Lyukshin(@gked), Microsoft
• Organization/project driving the design: Microsoft
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://www.chromestatus.com/feature/5745797339545600

Further details:

• I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): unknown
• Existing major pieces of multi-stakeholder review or discussion of this design: N/A
• Major unresolved issues with or opposition to this design: N/A
• This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

[kenchris]

Mouse has press and release events, but here you are only having a pressandhold, so you can not use it for pressing and dragging pen to select regions or text. Wouldn't it be better to have those instead?

[cynthia]

Thanks for bringing this to our attention. Including the comment above, here are the questions we have.

1. Why is the down/up pattern different for this from other (existing) input events? Pressandhold has limitations as @kenchris notes above.
2. Why is .connect() needed? Is this to ensure exclusive access to the pen, or a power saving measure?
3. It feels like having a "disconnect" event will be useful, since devices (as far as we are aware) automatically disconnect and an application should be able to know if it needs to call connect() again.
4. What happens when a page which is connected to a pen gets blurred? (e.g. user switches to a different tab)

[atanassov]

@kenchris and myself took another look at this issue during our "Kronos" VF2F. Given there hasn't been any traction on the comments left to authors above, we opened new issues in the Microsoft Edge Explainers repro and will review again once they have been addressed.

[BoCupp-Microsoft]

@kenchris @cynthia @atanassov thanks for your your patience. Your originally posted questions slipped my attention.

RE: this comment from @kenchris and question 1 from @cynthia:

Mouse has press and release events, but here you are only having a pressandhold, so you can not use it for pressing and dragging pen to select regions or text. Wouldn't it be better to have those instead?

This section in the explainer covers why:

Corresponding down/up events are not proposed because of hardware limitations, e.g. the Microsoft Surface Pen produces a signal describing the gesture, not the state transitions of the actual button.

RE: question 2 on the necessity of connect:

Why is .connect() needed? Is this to ensure exclusive access to the pen, or a power saving measure?

The reason is explained in the System Interactions section:

Note on some operating systems, pen events may trigger a default action. For example, on Windows, the shell provides configurable behavior for launching frequently used inking applications when pen button interactions are detected. Applications may register with the system to provide alternative behaviors and suppress the default behavior from the OS.

For web applications, only active windows that have a connected PenEventTarget can override the system behavior. As a result, any internal registration required to ensure that a web app receives the proper events should take place any time the active window changes, or the set of connected PenEventTargets change for the active window. The set of events to register is the unioned set of all events described by the PenEventDescriptors of the connected PenEventTargets for the currently active window.

In Windows you can configure the behavior of the pen button events and opt to let apps override those behaviors. The configuration UI looks like this:

Calling connect on a PenEventTarget will cause the web page to register the PenEventTarget's PenEventDescriptors with the system so that the web page can take over the events from the shell.

RE: question 3 on the need for a disconnect event:

It feels like having a "disconnect" event will be useful, since devices (as far as we are aware) automatically disconnect and an application should be able to know if it needs to call connect() again.

The response to question 2 may have already made the purpose of connect more clear: connect isn't connecting to the pen, its registering the intent of the app to override system behaviors. If the pen's bluetooth connection is interrupted, the web app doesn't need to call connect again.

RE: question 4 on blurred web pages with connected PenEventTargets

What happens when a page which is connected to a pen gets blurred?

I tried to cover that under the second paragraph of System Interactions

For web applications, only active windows that have a connected PenEventTarget can override the system behavior. As a result, any internal registration required to ensure that a web app receives the proper events should take place any time the active window changes, or the set of connected PenEventTargets change for the active window. The set of events to register is the unioned set of all events described by the PenEventDescriptors of the connected PenEventTargets for the currently active window.

Basically only PenEventTargets for the active window will receive events. On systems like Windows, the user agent must take care to return control to the shell when the active window changes (tab switches or you switch apps).

[cynthia]

@BoCupp-Microsoft. apologies this took so long. @atanassov and I discussed this during our F2F.

Corresponding down/up events are not proposed because of hardware limitations, e.g. the Microsoft Surface Pen produces a signal describing the gesture, not the state transitions of the actual button.

We're not really sure designing an API around a hardware limitation is a good idea. Feels like it would make sense to design a generic API and produce synthetic events for cases where hardware limitations disallow implementing certain features. (e.g. like how touch -> mouse was done)

Additionally, re-reading the explainer, it also seems like Pointer Events is being used to communicate button presses? In that case what are the three special button events for? Having to detect button presses in two different ways seems unintuitive.

Re. the connect() behavior, this is Windows-specific behavior that is being superimposed onto the web and I don't think that's something we are comfortable with. Have you discussed this with other implementors?

[kenchris]

@BoCupp-Microsoft any update to our comments?

[kenchris]

@darktears as you are on top of the USI standard, does that cover any of this?

[darktears]

I agree with @cynthia that it sounds weird to have a separate set of events for button presses when the Pointer Events deliver those (and if they don't fit the bill, we should expand these). I believe the reason Microsoft went this way is because the Surface Pen delivers its events not only through the digitizer but also with bluetooth when not in range with the screen (for the buttons/gestures they're talking about). And I'm assuming that when not in range the events are probably not coming through standard window message (WM_POINTER_DOWN etc) thus why they thought on an additional API because the underlaying API requires to create a separate connection.

Today as far as I can remember in Chromium, Pointer Events are delivered when the pen/stylus is able to communicate with the digitizer but that's an implementation limitation, I don't think it's written anywhere that pointer events could not be dispatched even if the stylus isn't in range with the digitizer. Now of course if we start to dispatch pointer events just for button state changes it begs the question of what would be the values of the other Pointer Events attributes provided we have no way to fetch them until the stylus is in range with the digitizer (e.g. position, tilt, pressure etc). Maybe we could add an attribute to the events to represent whether the stylus is in range with the digitizer. This would make quite clear that the values depending on being in range will be set to 0. That way you could still propagate the button state changes.

[BoCupp-Microsoft]

@cynthia and @darktears, regarding the following:

I agree with @cynthia that it sounds weird to have a separate set of events for button presses when the Pointer Events deliver those (and if they don't fit the bill, we should expand these).

PointerEvents don't seem like a good semantic match for pressing a button on a pen. Under the alternatives considered section of the explainer both the "Dispatching Additional PointerEvents" and the "Dispatching of Existing PointerEvents" sections cover why. PointerEvents relay some interaction with the screen. PenButtonEvents are screen independent and relay information about interaction with buttons on the pen, not contact (or near contact) with the screen.

We're not really sure designing an API around a hardware limitation is a good idea...
Re. the connect() behavior, this is Windows-specific behavior that is being superimposed onto the web and I don't think that's something we are comfortable with.

It doesn't appear to me that the hardware limitation or the need to override system behaviors for a pen interaction are unique to Windows. Exhibit A (for Apple): The Apple pencil seems to have similar APIs that apps could handle for the double-tap gesture (which by default switches tools or does something else as configured in iPadOS settings). I could see the Apple pencil exposing its double-tap gesture to the web and signaling that its been overridden by the web app via the proposed dblclick event and connect API.

Similarly, it looks like Chrome OS also generates a unique key event (like Windows) for a Bluetooth double-click button on a stylus as mentioned in this article and this related commit. I could also see the proposed dblclick event and connect API being a good fit here if the OS wanted to allow authors to override the built-in behavior like iPadOS and Windows can.

Have you discussed this with other implementors?

It looks like we did make a request for position here and that Apple and Mozilla are trending towards not interested. I'd like to circle back with them to see if that's the their final word on the matter. The primary concern seems to be that this might be too hardware-specific, yet I see multiple platforms that all have similar capabilities that could be made available to the web through this API. I'm not sure if that was well understood at the moment we originally released the explainer and those comments were made.

[cynthia]

Thank you for getting back to us, and apologies this took so long. While we aren't entirely satisfied with the outcome of the discussion, we also would prefer not to block work from happening. If you have future iterations which warrants a second round of reviews, feel free to request us to reopen. Thank you for bringing this to our attention.

[BoCupp-Microsoft]

Thanks for your consideration and comments.