Cursor pointer API

[okwasniewski]

This PR describes a new cursor: "pointer" API for iPad and Apple Vision Pro.

I've created this proposal together with @Saadnajmi

A rendered version of the proposal can be read here

[NickGerleman]

@javache shared that Vision Pro SDK does not currently allow tracking gaze, beyond declarative hover events, to protect privacy of what the user is looking at. Under that model, pointer events don't seem possible.

[Saadnajmi]

To clarify, are you saying the existing pointer events API in React Native won't work in visionOS? The new hover effects that visionOS added seemed to me something separate from the existing APIs, which I thought should still work without issue?

[NickGerleman]

Vincent left more comprehensive information about this below.

[NickGerleman]

Having unprefixed, single platform props for shared components like Pressable is a little bit smelly.

There are various schemes for prefixing and suffixing platform specific props in RN, and also ones we can do that we haven't tried yet (<Pressable apple:hoverEffect="lift">)

This might actually belong on the Platform's View (or VisionOSView instead), if we would ever want to support hover style outside Pressables.

[Saadnajmi]

In my head, something like hoverEffectIOS or hoverEffectApple is fine, though I feel it has been somewhat inconsistent in the past ( elevation isn't elevationAndroid, etc..). As implemented in the visionOS fork right now, it is a prop on the base <View>, and <Pressable> just sets a default that isn't none.

[matthargett]

I’m curious, what do you call these events for Windows Ink (Surface or Asus Pen)? Or for hand selection tracking in MRTK/HoloLens? I think there’s both prior art and probably a way to (finally) pull some of these offshoots forward in a unified way.

[Saadnajmi]

I have no idea what windows would call it 😅

[NickGerleman]

RN Windows solution for this is pretty crummy.

The event shape for custom onMouseEnter, onMouseLeave is effectively exact copy of PointerPoint

So, API is not coherent with other platforms. IIRC it may not even be typed.

[NickGerleman]

Pointer events unifies all of that, for I guess everything but Vision OS requiring declarative handling.

[NickGerleman]

RN on Quest ends up doing something like:

1. Set declarative information on view
2. In native, on pointer move, show underlay, vibrate controllers, etc on native hit test, before ever dispatching to JS

Some of this is exposed via Animated.event, which is a facility to animate native props when firing en event, before delivered to JS.

[vincentriemer]

Thank you so much for putting together this RFC! I’m more or less going to take this opportunity to elaborate on my own alternative to this proposal that I’ve been thinking about for a while now, but didn’t have the bandwidth/priority to put one together myself.

The summary/motivation you’ve described in this RFC is extremely similar to what I was thinking but there are big differences in how we approach the API design. As additional context I am part of the XPlat React/Convergence workstreams focusing on interoperability between platforms including web, and that greatly informs the lens I look through when approaching these APIs.

As the person who is working on implementing Pointer Events, I was immediately curious as to the Vision Pro could take advantage of my work when it was announced. I quickly learned that the underlying eye gaze coordinates are never provided raw to userland code (unless an indirect/direct gesture is underway). This is instead done declaratively either through the APIs mentioned in the parent RFC or by default with Apple’s provided UI elements like UIButton — but this made me curious as to what Apple was doing for websites/Safari.

Turns out Apple may have done the API design for us already! In the WWDC 2023 session titled “Meet Safari for spatial computing” around 4:58 they address this gap by saying that Safari will provide the hover highlight effect 1) on elements with roles that suggest they’re interactive or — most importantly — elements with the CSS style cursor: pointer.

This is the API I think will best provide interoperability between all of the platforms I can think of, and makes the introduction of a new API to the core more palatable as its no longer specific to Apple platforms. Using cursor: pointer will be a clean match with the web and there are cursor APIs on Android and (I’m guessing) Windows/MacOS.

The only rub with this API design is that Apple’s API doesn’t just have one UIHoverEffect that is used, it has multiples and you’ve addressed that in your RFC but mine currently doesn’t. I need to spend the time to do more verification on the Vision Pro’s simulator but I’d guess that Safari assigns the interactive elements on a web page the UIHoverAutomaticEffect and that’s what I believe that React Native should map cursor: pointer too. Here’s a quick video of a prototype I had made taking this approach.

As for the other, more specific effects I think those should be platform specific and we should use “prefixed” values like the web does (at least they used to). The core can consider the cursor StyleSheet property to be a string value and if there are specific cursor styles that don’t fit into the keywords enumerated in the web specification the platform’s underlying view should listen for their own, platform prefixed values (and other platforms can ignore/no-op on values they don’t handle). As an example: on Apple platforms the two explicit, platform specific hover styles are lift and highlight so I think that if we want to expose them as an value for cursor they should be exposed as -rn-apple-lift and -rn-apple-highlight.

To begin with though (and to keep the scope of this RFC small) I’d suggest to at least start with just the cursor: pointer use-case and we can address the platform-specific values later as that as a separate RFC would probably include discussions on auxiliary gaps such as defining multiple/fallback values or having functionality equivalent to @support queries.

[Saadnajmi]

@vincentriemer Thanks for the detailed response! I spoke with @okwasniewski offline and I think we both agree that pointer:cursor mapping to the automatic hover effect (or whatever Apple is doing for web) makes sense! A couple of extra thoughts:

1. React Native macOS (with help from Infinite Red and Meta's Messenger desktop team) previously implemented the cursor style prop, using as much of the W3C names as possible: Add cursor prop to View and Touchables microsoft/react-native-macos#760. If the implementation is fine, maybe the approach we could take is to upstream part of that PR and combine it with Oskars' visionOS/iOS implementation. A caveat is that macOS supports many more cursors than iOS/iPad do, and I have no idea what Android does. Therefore, I think leaving the cursor prop as a string makes sense, with the caveat that maybe only the value pointer is supported in React Native iOS, while other platforms (like macOS/Windows/Web) may support a whole lot more? I don't know how far along your branch / draft is, I'm curious what you think of that?

2. For macOS, cursor: pointer adds the pointing hand cursor that I see used on web, but not actually in native Appkit experiences much. iPad also sparingly uses hover/pointer effects, I don't think they are intended to be on every button. For that reason, I think it might make sense to leave the prop off by default on all platforms, rather than my original thinking of having Pressable set cursor: pointer by default.

[okwasniewski]

Thanks, @vincentriemer @Saadnajmi @NickGerleman @matthargett for great feedback on this RFC. I've updated the proposed solution as @vincentriemer described in his comment.

[Saadnajmi]

This would require some refactoring, but it would be nice if we could support the full custom border via separate-corner-radiuses-per-corner that RN supports (see RCTBorderDrawing.h). This is easier to do on Fabric than on Paper I think, but should be doable with both.

[Saadnajmi]

I'd be curious to connect a mouse to an Android phone or tablet and see if the pointer ever changes around the system UI. I know you get a mouse cursor, I can test later. Regardless, this RFC should be apple-only to start IMO

[vincentriemer]

Android has a cursor that's more similar to web than Apple platforms via this method on View. I'd rather limit the RFC to apply to all platforms (at least theoretically) but limit the APIs surface area (like focusing on just pointer: cursor instead of all the different types).

[vincentriemer]

1. React Native macOS (with help from Infinite Red and Meta's Messenger desktop team) previously implemented the cursor style prop, using as much of the W3C names as possible: Add cursor prop to View and Touchables microsoft/react-native-macos#760. If the implementation is fine, maybe the approach we could take is to upstream part of that PR and combine it with Oskars' visionOS/iOS implementation. A caveat is that macOS supports many more cursors than iOS/iPad do, and I have no idea what Android does. Therefore, I think leaving the cursor prop as a string makes sense, with the caveat that maybe only the value pointer is supported in React Native iOS, while other platforms (like macOS/Windows/Web) may support a whole lot more? I don't know how far along your branch / draft is, I'm curious what you think of that?

I think to start with we should say that cursor: pointer and cursor: auto (the implicit value when cursor is unspecified) are the only values officially supported by the core but other platforms can support additional values they choose — though I'd add the caveat that unless the extra value that platform wants to support is listed in the W3C spec, they should prefix the value to avoid any naming/platform conflicts in the future.

2. For macOS, cursor: pointer adds the pointing hand cursor that I see used on web, but not actually in native Appkit experiences much. iPad also sparingly uses hover/pointer effects, I don't think they are intended to be on every button. For that reason, I think it might make sense to leave the prop off by default on all platforms, rather than my original thinking of having Pressable set cursor: pointer by default.

Agreed. I do think it's worth exploring putting cursor: pointer on Pressable by default but it's going to be potentially messy and ultimately out of scope of this RFC.

[okwasniewski]

@vincentriemer I agree that limiting this to cursor: "pointer" and cursor: "auto" and instead having this on all platforms is a better option. How can we move forward with this RFC? I could prepare a PR for iOS side of things. I could also try Android but I am not that familiar with it

[Saadnajmi]

Update: Ready for review!
I've started working on an implementation of this RFC in both React Native (for just iOS) and React Native macOS (for visionOS, since macOS already had a cursor prop). Still rough but wanted to have my working branches up as we're already getting asks for this prop internally.

[vincentriemer]

lgtm

[okwasniewski]

Hey! As this got implemented in facebook/react-native#43078 I think this PR can be merged (?)

[Saadnajmi]

While we're on the subject, are there opinions against extending the "pointer-events" prop (not the new w3c api but the old one that controlled touch events) to also apply to cursors? That particular would help with some edge cases related to nested cursors, but I also know it's an old prop and may not be one we want to reuse. I think can be outside the scope of this rfc since the initial implementation already merged.