cursor should not trigger mouseenter when in VR

[diarmidmackenzie]

Description:

• A-Frame Version: 1.5.0
• Platform / Device: Quest 2, Pro or 3

When building an experience that can operate in desktop or VR mode, it's common to build desktop interactions using a cursor, but then use a different interaction system in VR (controller-based or hand-tracking).

In VR mode, we don't expect the cursor to fire a "mouseenter" event, but sometimes it does.

The reason is that the raycaster associated with the cursor remains in the scene on entering VR, pointing in whatever directon the last mouse pointer position was detected prior to entering VR.

If this ray intersects an object, the raycaster will fire a raycaster-intersection event, which will lead to cursor emitting a mousenter event here

This can trigger unexpected mouse hover effects to appear in applications that haven't anticipated for the possibility of receiving a mouseenter event in VR mode.

I'm not entirely sure how to fix this. First idea was to check for the combination of rayOrigin: mouse & vr mode, in setInteraction() and do an early return

if (this.data.rayOrigin === 'mouse' && 
    this.el.sceneEl.is('vr-mode')) return;

However there may be some cases where a mouse is a valid input mechanism in VR mode. E.g. on desktop if you enter full screen, you are "in vr" (this.el.sceneEl.is('vr-mode') returns true) and yet mouse remains a valid input mechanism.

I don't think there's a simple query we can do to determine whether we are on a platform where mouse input is valid or not. In the absence of such a query a fix might be:

• on entering VR, set a flag this.mouseAvailable to false
• on detection of a mouse move or click event, set this.mouseAvailable to true
• use this.mouseAvailable to determine whether ot not to so an early return on detection of a raycaster intersection.

I'll draft a PR based on this concept, as a basis for further discussion.

[dmarcos]

However there may be some cases where a mouse is a valid input mechanism in VR mode. E.g. on desktop if you enter full screen, you are "in vr" (this.el.sceneEl.is('vr-mode') returns true) and yet mouse remains a valid input mechanism.

Not sure that's a very common use case or even makes sense in immersive mode (no 2D screen available)

if (this.data.rayOrigin === 'mouse' &&  this.el.sceneEl.is('vr-mode')) { return; }

Looks reasonable. If people wanna use mouse in VR we can try to accommodate later.

[diarmidmackenzie]

One use case I already have is for "VR + mouse" (which would be broken by the naive fix) is testing VR applications on desktop.
https://diarmidmackenzie.github.io/aframe-components/components/desktop-vr-controller/

These days we also have Meta's WebXR emulator chrome extension, but I personally still prefer manipulating the controllers directly in full-screen mode, over manipulating them in a separate window.

[dmarcos]

How do you manipulate the controllers in fullscreen? In VR mode you mean?

[diarmidmackenzie]

Sorry I could have explained more clearly.

If you open this example on desktop (with no VR headset attached), and enter VR (which switches to full screen, as there is no headset) then you can use the mouse (and keys) to manipulate the controllers.

This relies on the fact that even though A-Frame thinks we are in "VR" mode, we are actually just in full-screen on desktop, so the mouse is still available, and cursor events still fire.

[dmarcos]

I would say this.el.sceneEl.is('vr-mode') or this.el.sceneEl.is('ar-mode') should be false in fullscreen mode so can disable mouse cursor as you mentioned.

We should probably revisit the enter immersive mode flow. Some legacy there. Starting with probably the method enterVR in a-scene. I would probably rename to sceneEl.enterImmersiveMode('vr' or 'ar' and maybe 'fullscreen' but not sure)

[dmarcos]

Fullscreen and this.el.sceneEl.is('vr-mode') is remanent of cardboard

[diarmidmackenzie]

It may be a remnant of cardboard that's been left in by chance, but it's enabled some really handing debugging flows.

E.g. if I have an entity that changes it's appearance based on is('vr-mode') it has been really convenient to be able to test and debug this on desktop, without having to put a headset on.

Personally I'd be disappointed to lose that capability.

[dmarcos]

I think the nomenclature and language used makes little sense now. We should clean it out. There's the emulator and other alternatives that are better and don't add maintenance costs and make the API less clear or ambiguous. I'm sure there's a way to still mimic your workflow without it. We can add add a new state is('fullscreen') that it's more descriptive.

The emulator can also be improved. It could for example get events from the main viewport to move the controllers

[dmarcos]

To make sure I understood correctly. You wanna keep cursor events while in fullscreen, state named now as vr-mode for your testing workflow?

[diarmidmackenzie]

Yes, that's a clear summary of what I'd like to keep.

But on further reflection, I don't think that should be a priority if it's preventing you from getting to a clean consistent interface for A-Frame. Focus on getting the A-Frame interfaces right, and I'm sure I can monkey patch or mock things to make desktop-vr-controller continue to work, if I decide it's worth keeping.

SImilarly for the desktop-based is('vr-mode') testing above, it should be easy enough to build a custom component that simulates VR mode even though on desktop.

No need to pollute A-Frame core with any of this.

[dmarcos]

Opened this meta-quest/immersive-web-emulator#54

Mouse / keyboard control of headset and controllers is clearly useful.