Simplified V1 Input Proposal

[toji]

Background

A good chunk of this is, primarily, a re-focusing of the previously produced Input Explainer. As a result nothing here should really be surprising.

For those who don't already know, we are unable to continue with the design put forward in that document because of concerns about compatibility with upcoming (and as of this writing unreleased) VR/AR standards. I will not dive into a comprehensive evaluation of those incompatibilities, due the said unreleasedness of the standard in question, but will broadly note that it's not known at this time if it will allow the full input device state to be queried in the manner the previous explainer would require.

Given that, and given that we would prefer to have users begin using the WebXR Device API as soon as is reasonable without being blocked on third parties, I propose that we re-focus on exposing a minimal but broadly compatible subset of the previously discussed functionality and have some clear ideas of how it could evolve to fit a variety of underlying input systems in the future.

Requirements

The "simple" proposal from the previous explainer was one that just allowed developers to listen for basic point-and-click events from an source of VR input, which is enough to enable basic button-based UIs. This is "good enough" for video players, galleries, some simple games, etc. It is insufficent for more complex uses like A-Painter style art apps, complex games, or really anything that involves direct manipulation of objects.

That's regrettable, but a limitation that I feel is worth accepting for the moment in order to enable the significant percentage of more simplistic content that we see on the web today.

So, what we need to enable that level of input is:

• An event that fires when the input's primary interaction method occurs. Important that this be treated as a user activation event.
• Frame-to-frame tracking of the input ray

I would also propose that, since this would be all we offer initially, we make this just a teensy bit more useful and future-proof by adding:

• Notifications of starting and ending the primary interaction method. (ie: Button down/button up)

This would allow a bit more nuance in the interactions allowed by the system, giving the option to drag items around, for example.

Proposal

I find it easier to talk about these things when looking at an interface, so I'll start with a proposed IDL:

enum XRHandedness {
  "",
  "left",
  "right"
};

interface XRInputSource {
  readonly attribute VRHandedness handedness;
};

interface XRInputPose {
  readonly attribute Float32Array? gripMatrix;
  readonly attribute Float32Array? pointerMatrix;
};

//
// Extensions to existing interfaces
//

// Aside: I really think we should consider renaming this to just XRFrame
partial interface XRPresentationFrame {
  XRInputPose? getInputPose(XRInputSource inputSource, XRCoordinateSystem coordinateSystem);
};

partial interface XRSession {
  attribute EventHandler onselect;
  attribute EventHandler onselectstart;
  attribute EventHandler onselectend;

  attribute EventHandler oninputdeviceschange;

  FrozenArray<XRInputSource> getInputDevices();
};

//
// Events
//

[Constructor(DOMString type, XRInputSourceEventInit eventInitDict)]
interface XRInputSourceEvent : Event {
  readonly attribute XRPresentationFrame frame;
  readonly attribute XRInputSource inputSource;
};

dictionary XRInputSourceEventInit : EventInit {
  required XRPresentationFrame frame;
  required XRInputSource inputSource;
};

Tracking and rendering

Let's dive into tracking first, since it's relatively straightforward. xrSession.getInputDevices() returns a list of any tracked controllers. This does not include the users head in the case of gaze tracking devices like Cardboard. By themselves these objects do basically nothing useful.

On each event the user can iterate through the list and call xrFrame.getInputPose(inputDevice[i], frameOfReference); to get the pose of the input in the given coordinate system and synced to the head pose delivered by the same frame. This can be used to render some sort of input representation frame-to-frame. (Note: I'm not including anything that describes a controller mesh for practicality reasons. We can investigate that later. In the meantime apps will just have to use app specific or generic resources.)

The input device would be rendered using the gripMatrix, as that's what should be used to render things that are held in the hand.

Pointers are a little more subtle. We want to render a ray coming off the controllers in many cases, but not the users head. However if the device is gaze based we do still want to draw a gaze cursor, and if the session is a magic window context we don't want to draw any cursor at all. So a bit of logic is needed to handle that. When pointers are drawn they should be drawn using the pointerMatrix, which may differ from the gripMatrix for ergonomic reasons.

The basic pattern ends up looking like:

function drawInputs(xrFrame)
{
  let inputDevices = xrSession.getInputDevices();

  if (inputDevices.length) {
    // If input devices are reported always draw them with pointer rays/cursors.
    for (let inputDevice of inputDevices) {
      let inputPose = xrFrame.getInputPose(inputDevice, xrFrameOfRef);
      if (inputPose) {
        drawAController(inputPose.gripMatrix);
        drawAPointer(inputPose.pointerMatrix);
        drawACursor(inputPose.pointerMatrix, cursorDistance);
      }
    }
  } else if (xrSession.exclusive) {
    // Render a gaze cursor for exclusive sessions with no input devices.
    let devicePose = xrFrame.getDevicePose(xrFrameOfRef);
    drawACursor(devicePose.poseModelMatrix, cursorDistance);
  }

  // Render nothing for non-exclusive sessions with no input devices.
}

I'd expect that we'll get a Three.js library real quick that adds simple controller visualization to your scene and does all the right things in this regard.

Primary input events

Handling primary input events is the other half of this proposal. A quick recap of what that means, copy-pasted from the previous explainer:

The exact inputs that trigger these events are controlled by the UA and dependent on the hardware that the user has. For example, to trigger a "select" event on a variety of potential hardware:

• On Daydream, the user would click the controller touchpad.
• On HoloLens, the user would perform a tap with their index finger or say the system "Select" keyword.
• On a Vive controller, Oculus Touch or Windows MR controller, the user would pull the trigger.
• On Cardboard the user would press the headsets button.

To listen for any of the above the developer adds listeners for the "select", "selectstart", or "selectend" events. When any of them fire the event will supply an XRPresentationFrame that's used to query input and head poses. The frame will not contain any views, so it can't be used for rendering. It also provides an XRInputSource that represents the input device that generated the event. This may be one of the devices returned by xrSession.getInputDevices() (in the case of a tracked controller) or one that's not exposed anywhere else (in the case of a headset button, air tap, or magic window touch)

xrSession.addEventListener("select", onXRSelect);

function onXRSelect(event) {
{
  let inputPose = event.frame.getInputPose(event.inputSource, xrFrameOfRef);

  if (inputPose) {
    // Ray cast into scene with the pointer to determine if anything was hit.
    let selectedObject = scene.rayPick(inputPose.pointerMatrix);
    if (selectedObject) {
      onObjectSelected(selectedObject);
    }
  }
}

The exact interpretation of the pointer is dependent on the source that generates the event:

• For tracked controllers it's a ray originating at the tip of the controller.
• For gaze cursors it's a ray originating at the center of the users head and pointing in the direction of their gaze.
• For magic window clicks the ray would originate at the graphics near plane and project out from directly under the cursor/touch point.

Use cases

The above capabilities give developers enough to handle the following (non-comprehensive) scenarios:

• Dance Tonite-style passive viewing experiences
• Video players (clicking buttons)
• Image Galleries (clicking to expand, click and drag to scroll. No touchpad scrolling)
• Matterport-style navigation (click to teleport)
• Partial SketchFab-style use (click to teleport, but no touchpad scaling)
• Simple shooter games (would need to use in-world buttons to switch weapons)
• Simple painting apps (would need to just assume presence of 6DoF controllers, tools changes would need to use in-world UI)

Obviously we'd like to enable more robust usage, but this does allow a pretty wide range of apps in the most broadly compatible way we can manage.

Future directions

So that's the extent of the current proposal, but it's good to have an idea of how we could extend it in the future. A few thoughts on that:

The current Gamepad API maintainers would like us to continue using it in conjunction with VR, and have expressed a willingness to refactor the API if necessary to make it more generally useful. If we wanted to go that direction (and were confident we could map it to all relevant native APIs) I would propose that we either expose Gamepad objects on the XRInputSource or make XRInputSource inherit from Gamepad (We would drop the pose extensions and displayId).

But if that's not practical, which is a very real possibility, my general line of thinking is to add a way to query inputs by name or alias to receive back an object that can be used both for state polling of that element and input event listening. Something like this:

interface XRInputAction extends EventTarget {
  attribute EventHandler onchange;
  attribute EventHandler onclick;

  readonly attribute boolean pressed;
  readonly attribute boolean touched;
  readonly attribute double  value;
  readonly attribute double? xAxis;
  readonly attribute double? yAxis;
};

partial interface XRInputSource {
  XRInputAction? getAction(DOMString action);
};

interface XRInputActionEvent : Event {
  readonly attribute XRPresentationFrame frame;
  readonly attribute XRInputSource inputSource;
  readonly attribute XRInputAction inputSource;
};

Which could then be used like so to get the same effect as the "select" event documented earlier.

let selectAction = xrInputSource.getAction("select");
if (selectAction) {
  selectAction.addEventListener("click", onXRSelect);
}

[TrevorFSmith]

Moving the interpretation of common higher level semantic events down into the user agent is clearly necessary to support the creation of XR apps that will work across the many existing platforms and going forward on new platforms.

One thing that's not clear to me is how we pick the small set of initial semantic events and how we extend that set in the future. How do we draw the line between the initial set of events and the events that also feel somewhat inevitable, like secondary selection?

[kearwood]

This is looking really good -- I feel it captures well most of the concepts from our earlier iterations and meetings.

I have just a couple small suggestions:

1. Add an onselectcancel event handler.

In the case when a user begins a gesture (ie pressed a button down) and the content loses VR focus or positional tracking before the button is released, the user likely does not want the gesture to be acted upon. Rather than emitting a onselectend event, the browser could emit the onselectcancel event. Alternately, we could still fire onselectend, but include a "reason" attribute.

2. Could we include an "onback" event as well?

"out", "back", "undo", "exit", "home", "cancel" seem to be the second-most common interactions after "select" and could be handled by this event. Unlike select, this one would not have a raycast but would have onbackstart, onbackend, and onback events. Having start/end for this would enable long presses to go "home" while short pressed to go "back" for example.

[dmarcos]

It looks good to me. @toji How do you see the proposed XRInputSource API evolving to accommodate input data beyond buttons and axis like for instance a fully tracked hand or body suit?

[kearwood]

If the controller does not have any spare physical buttons to represent the "onback" event, it could recognize this in other ways, enabling consistency across all WebVR experience. Examples:

• shake the controller rapidly
• swipe on a trackpad
• double-click the "select" button or double-tap the trackpad

[toji]

@TrevorFSmith: This is probably the question that's caused me more stress than anything else lately. The most straightforward answer I've come up with is that if we're concerned about the divide we can jump directly to the getAction approach I outlined in the "Future directions" section and simply start with only one action. In order to feel confident about that approach, though, I think we'd want to be confident that it was going to be the solution long term, and I'm simply not yet. That could change with some API iteration.

I brought this up with @bfgeek prior to posting this issue, and his suggestion was just to do the simple, straightforward thing for the moment and not overcomplicate things. "Primary action" is a fundamental enough concept that if it becomes the sole input boondoggle against a more robust future system it won't look too weird.

@dmarcos: My gut tells me that body suits are not applicable here. For one, there's no hardware in wide use that does it yet, and thus the interfaces to it are undefined. You could speculate it looks something like a Kinect skeleton, but that may not be the end product for a variety of reasons. Two, even if you have full body tracking it's unlikely that developers want to treat a full body as equatable to existing controllers, which are almost always tracking hands in one form or another. I can envision a system where XRSkeleton and XRInputSource live alongside eachother, feeding off of the same underlying data: One for tracking the user's full pose and one that abstracts their hands into a more traditional pointing device for compatibility.

As for hand tracking, which I view as a more specialized case, we did talk about that in the context of the previous explainer and the hope was that we could attach a nullable hand skeleton directly to the XRInputSource when applicable, which could feasibly represent either Leap Motion-style visual tracking or Oculus Touch-style pose estimation. Doesn't seem appropriate for a v1 feature, but I do like having an idea of where that data could be attached.

@kearwood: +1 to the idea of communicating that the input was canceled somehow, though I'm curious if it applies in any situation other than focus lost? Can't think of any at the moment. Also, I'm hesitant to create a bunch of separate events, I kind of like your suggestion for adding a reason, though I'd maybe say take it a step further and have a more general selectchange event that includes the current state. (That's edging closer the the "Future directions" suggestion, not sure how far down that rabbit hole I want to fall.)

As for things like back/home/exit/etc I'm more conflicted. They absolutely make sense for browsing mode, but in that case the UA should handle it entirely. In WebXR content it seems like a lot of them are going to be context-sensitive and we don't have a reasonable way of knowing the context at this time. Certainly we can't be guaranteed that there will be enough discreet buttons to cover all of the desired functions, and if we enforce certain gestures we end up limiting the apps input possibilities. For example: I'd hate to prevent any games that simulate swordplay because all of that rapid slashing was interpreted as a back gesture.

So I think that until we start dipping our toes into declarative land we should ensure that the developer is fully in control of how they interpret inputs and we simply surface them in as consistent a manner as possible. That said, making it easier to recognize certain common gestures (touchpad swipes, double clicks, etc) seems like a unquestionably good idea. I'm not sure if the browser can provide anything there that a well built library can't, though. (This is, of course, assuming we have made more of the input state available to the page)

[AdaRoseCannon]

This looks really good.

For the controllers you get a Matrix but if the controller only has rotation capabilities should the browser return only a Matrix with rotation component?
Perhaps it should estimate position components based upon a shoulder/arm rig once it knows which hand the user is holding the controller in, what are your thoughts on this?

[toji]

@AdaRoseCannon: I agree that an arm model should implicitly be part of the matrix returned for 3DoF controllers. The only caveat is that we should ensure that the arm model doesn't alter the rotation of the controller, only it's offset. That way developers who care exclusively about the rotation (driving games, etc.) can trivially zero out the translation components and be left with an accurate orientation. We've confirmed that this is how the Daydream arm models work, and I would imagine that GearVR is the same.

That said, we should probably indicate when the position is emulated rather than sensor-based. That was in the original explainer but I took it out here for simplicity and because I hated the name. 😉 Maybe xrInputSource.positionEstimated?

As for which side the user's holding it on I'd expect the platform to know what the preferred hand is, so we should be able to just use that.

[sompylasar]

Looks good! I think the getAction approach is the most extensible and should be the only way to receive interaction events from the inputs. The method can in the future return more than one type of actions (interfaces) to allow variety of types of input with incompatible sets of attributes without requiring to extend the core input source interface.

What I'm concerned about is how this proposal addresses the input mode where the input is continuous, not a sequence of discrete events (actions, gestures).

[kearwood]

@toji re: +1 to the idea of communicating that the input was canceled somehow, though I'm curious if it applies in any situation other than focus lost?

Some other situations:

• controller moved out of tracked volume / safety-guardian space
• User pressed button on second controller, pointing at different target
• 6dof controller lost tracking
• HMD lost tracking
• HMD was removed during gesture
• Another non-webvr gesture was recognized using the same physical controls (ie. very long press of button or additional button pressed at the same time)
• Lost communication with controller (ie. wireless interference, battery depleted)

[AlbertoElias]

Really like this. Big +1 to the arm model, and as long as v1 is extensible to all possible buttons on an input device, I think it's great

[joshmarinacci]

I like this proposal. I'm fine with adding it to the GamePad API, though that API needs some work as well (which I've been digging into lately). I'm confused about how the developer can tell which sort of device the user currently has, and therefore if they should draw the gaze cursor and other interaction overlays. Does the developer look for the existence of the gripMatrix vs pointerMatrix objects?

[toji]

I'm confused about how the developer can tell which sort of device the user currently has, and therefore if they should draw the gaze cursor and other interaction overlays.

This was definitely under-specified in my above text, and I'm not sure I'm totally satisfied with the implied proposal I made.

The code sample shows that if there's no tracked controllers then the device is implied to use a gaze cursor. That seems to fit for Cardboard-style devices and probably HoloLens uses (even though that does provide limited hand tracking, it doesn't expose hand position frame-to-frame AFAIK). However on further reflection this wouldn't really be appropriate for something like the GearVR's headset controls or the Oculus Remote. Those provide more than one-bit inputs, so presumably we'd like to expose them eventually, but are not actually tracked independently from the headset. I guess there's a few different attitudes we could take regarding those:

1. We should treat these like any other XR input device and expose them via xrSession.getInputDevices(), adding data to indicate their tracking capabilities (or lack thereof)
2. They're no different than a gamepad. We can expose them using the existing gamepad API and reserve xrSession.getInputDevices() for tracked devices only. select event could still fire.
3. That style of input is quickly disappearing, so we shouldn't consider it to be anything more than a fancy cardboard button.

[judax]

I like the proposal and agree with some of @kearwood comments to include the cancel event. Not sure about not using a different event handler though. I think it makes it more consistent (although I understand not wanting to swamp the API with handlers).

Some notes:

• If the type is InputSource (I prefer InputDevice, but I guess there is an argument to some types of inputs not being devices per se?), then the call should be getInputSources right?

• How will the app know what to render as the controller? For example, how does the app know that a Vive controller or an Oculus Touch controller should be rendered to match the type of device the user is handling? Or, as this is a high level abstraction that does not provide access to other buttons all the controllers should be rendered using a generic 3D model?

[NellWaliczek]

This is looking really good, @toji ! Thanks for pulling the key parts of the original proposal forward.

@judax yeah the intent with the naming is to also be representative of hand input (such as on HoloLens). Also, we had an optional glTF blob off of InputSource that maybe got missed somewhere in the shuffle. @toji , was that on purpose or can we just grab it and add it back in?

@kearwood As far as "cancel" goes, I agree that it's worth having!

It's great that we're taking a look at how we might integrate with gamepad going forward to help ensure our plan is solid. At the same time, let's be careful not to go too far down the road of designing the future solution when the whole point of this new propsoal is the address the fact that we aren't ready to do so yet ;)

[kearwood]

As per the Implementers' Call:

We should ensure that content can know when it should draw a controller or not, so that controllers are not drawn on top of your physical hands/controllers in an AR environment.