Representation of a rays in WebXR

[toji]

This came up on the AR-centric call this week: Conversations around the Hit Test explainer, which deals with rays quite a bit, have proposed representing them as two vectors: an origin and a direction. This is an easy to understand and use format, and is especially helpful if developers have to specify a ray manually.

The WebXR explainer, however, currently represents rays as a matrix, which indicates the transform to be applied to a ray with an origin at [0, 0, 0] and pointing down the -Z axis. The desire was to make it easy to render with, while still not making it too hard to get the origin/direction form through a couple of vector transforms.

Both approaches have pros and cons, but we should definitely pick one and use it consistently throughout the spec. In my opinion the following should be considered:

Origin/Direction form

Pros:

• Human readable
• Easy to build by hand
• Less values to communicate

Cons:

• When matrix format is needed it's harder to generate.
• Need to deal with non-normalized directional vectors. (Pretty minor)

Matrix form

Pros:

• Easy to use directly for rendering
• Producing Origin/Direction from a matrix is relatively easy
• Using it would mean WebXR only has to deal with one math primitive type.

Cons:

• Not human readable
• Difficult to construct by hand
• Overly expressive (rotation around z axis doesn't matter, scaling and shearing are unnecessary and detrimental)

My gut feeling, taking all that into account, is that using the origin/direction form is likely better for API usability long-term. It's unfortunate that it's more difficult to use for rendering, but if we ever expect users to provide rays back into the system (as the hit-test proposal does) then it seems overly complicated and error prone to ask them to compose and pass full matrices. (It's way easier to normalized the directional input implicitly than it is to try and handle a badly formed matrix.)

If we do go with origin/direction, I would suggest that we use an interface to represent it within the API. Maybe something like this?

interface XRRay {
  Float32Array origin;
  Float32Array direction;
}

or

interface XRRay {
  double originX;
  double originY;
  double originZ;
  double directionX;
  double directionY;
  double directionZ;
}

Both of which have their own issues in my mind.

[toji]

Bah, I just realized that this something of a duplicate of an issue on the hit-test repo: immersive-web/hit-test#8 Sorry!

[blairmacintyre]

I agree with this.

I would still like to decide how we deal with underlying systems, like ARKit, that do not allow arbitrary rays.

We could ignore it for now and "hope for the best": in the discussion over in the hit-test repo, I made a comment that it may be that subsequent versions of ARKit (and other APIs) eventually support arbitrary rays.

It is also unclear if Apple is ever going to implement WebXR, so this may be moot for the native implementations; for apps like our WebXR Viewer, I could see doing this:

• if the origin is at the camera, use ARKit's hit-test functionality
• if the origin is anywhere else, shoot a ray into the collection of surfaces ARKit has reported

If we add options to allow anchors to be created on surfaces or in the world where the hit occurred, this could also be implemented "manually" inside the implementation: it would require some book keeping, AND it would mean that certain kinds of hit testing might not be possible (i.e., ARKit allows you to request hits against world structure it knows about but hasn't reported yet). But, that seems ok.

[toji]

I share your concerns about the way that the native APIs are exposing hit-testing, but for the purposes of this specific issue it's likely to be moot. We will need to represent rays one way or another even in absence of hit testing.

[kearwood]

An alternative may be a quaternion plus an origin; however, I feel the simple 3-component origin plus 3-component direction is simplest to understand and convertable to other formats as needed.

[kearwood]

We may want to formalize what will happen if the ray direction is not normalized. eg, it will be normalized by the browser before the ray testing and no hit results will be made if the length of the vector is less than a minimal value.

[toji]

Follow up from today's call: Seems like everyone is in favor of a non-matrix representation for rays. The most obvious representation (origin, direction) appears to be the favorite due to it's easy-to-understand structure, but we should give at least some consideration to quaternions as Kip mentioned.

In terms of error handling (normalization, zero-length directions, etc) there's still some open questions but there was some support for the idea of having the UA normalize directions as they come in and simply failing to produce hits with zero-length directions as opposed to throwing an error. (A console warning may still be desirable.) No determination was made about how the presumed XRRay interface should be laid out.

[toji]

PR #357 is up now to address this.

[toji]

This is fixed now that the related pull request is merged.