Tag feedback: change from constants to enums + change XRHand into a map

[cabanier]

---

Preview | Diff

[cabanier]

Addresses #70 and w3ctag/design-reviews#568

[Manishearth]

/agenda to check with the group on this

[Manishearth]

@cabanier good start: a couple of things need to be done here:

• we need an explainer update
• we may need a readonly attribute XRJointName name on XRJoint?
• we should probably decide between iterable<k, v> and maplike. Are we allowed to define an iteration order for maplike? (cc @domenic). The key-value iteration as it is right now seems okay to me too but I'm sure people have preferences.

[cabanier]

• we need an explainer update

Certainly. I think we can do that after coming to a consensus on the new API

• we may need a readonly attribute XRJointName name on XRJoint?

I added it to XRJointSpace. Is there an XRJoint class?

• we should probably decide between iterable<k, v> and maplike. Are we allowed to define an iteration order for maplike? (cc @domenic). The key-value iteration as it is right now seems okay to me too but I'm sure people have preferences.

@domenic said that that was allowed so I added it as prose to XRHand

[Manishearth]

I added it to XRJointSpace. Is there an XRJoint class?

no, i just accidentally used the old name for it 😄

[thetuvix]

While the proposed terms here are more common English, I'm concerned that they are more ambiguous in practice:

• palm in OpenXR's hand tracking refers to a synthetic joint at the center of the hand that is useful for pressing with the whole hand. In WebXR, we decided to skip that palm "joint" - however, in this proposal that same term would be used to pose the metacarpal bones. Those joints do fall somewhere within the palm, but a developer encountering this naming in the enum would seem to have less context on how to correctly rig a hand model, which is the only real use case for metacarpal poses.
  
• base is ambiguous here - my first reading was that this was the name you chose for the metacarpal joints (the most "base" joint for a given finger), and I only later figured out that I had base and palm swapped. "base" was the non-anatomical term we considered for the metacarpal joint in our original native API, but we abandoned it due to this ambiguity.
• I am concerned that lower and upper have the same ambiguity - when my hands are on my keyboard right now, which joints are more "lower"? When a user is standing naturally, the joints that are lower will be opposite to how they are named in this PR. The terms "proximal" and "distal" are medical jargon, but they are not hand-specific, they refer to being closer to the core of your body (proximal) or farther from the core of your body (distal).

If we do want common names for the finger joints, we could consider knuckle1/knuckle2/knuckle3, as "knuckle" is a reasonably common English word for referring to the visible finger joints. However, when we considered that internally in our original native hand tracking API, some folks thought the knuckles would be numbered from base to tip and some assumed tip to base. That was the key reason we never moved forward with that approach.

Naming the joints after the bones their position and orientation will pose was the least ambiguous naming and gave the smallest wiggle room for a standard that apps will expect to behave equivalently across multiple vendors.

[Manishearth]

Yeah, I find the medical terminology to be clearer since we're not inventing new terms. For example, "base" can either mean the knuckle or the metacarpal (since when we say "index finger" here we're talking about all of the bones). Either way it's something people will have to learn if they want to deal with individual joints, it seems better to make them learn terminology that is well documented and consistent across other APIs.

Fun fact about "lower" vs "upper": Different people inspect their fingernails differently -- some put their palm out flat, others curl their fingers in with the palm pointed upwards. Not sure of how much that will affect people's perception of which end of the finger is "upper", but i can imagine some people will feel differently. The hand has rotational freedom, there isn't really an "upper" end. I'm slightly reminded of how jarring it is whenever I see 上 (usually, "up") in Mandarin used to denote "before"/"previous", since my brain has a strong implication that the future is "up".

[kdashg]

FWIW while I'm sympathetic to lay-intelligibility concerns, I find the priority for correctness and non-ambiguity in both implementation and uses of API inclines me towards using the medical names. While the terms aren't in common use, even for native English speakers, with repetition comes familiarity. Conversely, for people who don't need all the details and just code it once (or just use "tip"), this isn't too much of a concern.

Maybe a minor wrench in things is that we're providing info on the joints locations not the bones per se. (I think?) What this means is that "metacarpal location" is actually (it looks like) the base of each metacarpal, not the terminating point of the metacarpal. For e.g. the knuckles of a fist, you want the tips of the metacarpals, so you would ask for the (base joint of the) "proximal" bone. Likewise "distal" is the base of the distal bone, not the most distal point on the finger. So long as we establish a clear and consistent approach.

[Manishearth]

Maybe a minor wrench in things is that we're providing info on the joints locations not the bones per se. (I think?)

It's both! The orientation of the spaces is along the bone. These are called joint spaces but they're anchored to the bottom (closer to the wrist) of the correspondingly named bone.

[cabanier]

FWIW while I'm sympathetic to lay-intelligibility concerns, I find the priority for correctness and non-ambiguity in both implementation and uses of API inclines me towards using the medical names. While the terms aren't in common use, even for native English speakers, with repetition comes familiarity. Conversely, for people who don't need all the details and just code it once (or just use "tip"), this isn't too much of a concern.

The tag guidelines state:

API naming must be done in easily readable US English. Keep in mind that most web developers aren’t native English speakers. Whenever possible, names should be chosen that use common vocabulary a majority of English speakers are likely to understand when first encountering the name.
For example setSize is a more English-readable name than cardinality.

So I'm reluctant to go back to names that people will have to look up on wikipedia.
It seems people were objecting to the implied direction of the proposed names and base/palm. Since the vast majority of users will just ask for the finger tip, maybe we can keep the names for the finger tips and then start counting.
For instance:

• pinky-finger-tip = little finger tip
• pinky-finger-1 = little finger distal phalanx
• pinky-finger-2 = little finger intermediate phalanx
• pinky-finger-3 = little finger proximal phalanx
• pinky-finger-4 = little finger metacarpal

[Manishearth]

maybe we can keep the names for the finger tips and then start counting

I already spoke out against this in the issue: the problem is that not every hand tracking system exposes the same joints, so how you count is tricky. Especially around the thumb, where the choice of joints is rather variable. Furthermore, the direction of the numbers can be confusing, and folks will have to remember if it's 0-indexed or 1-indexed. This does not seem to be solving the problem, people will still have to refer to a diagram.

So I'm reluctant to go back to names that people will have to look up on wikipedia.

My contention is here, no matter what we pick, they will have to look up a diagram on some docs (MDN, etc), so the best we can do is pick something without ambiguity.

The core issue here is that people do not, on a day to day basis, talk about hand joints. There is no day-to-day name for the individual joints. If one had to communicate something about specific joints, one would probably point at the relevant joints, and if pressed to name them, different people would say different things. I don't see an easy, unambiguous way of naming the joints purely textually in plain English. Numbering them doesn't work because it's unclear which is "1", and which joints are included. Naming them based on relative position needs a reference point for thing to be relative to ("upper" and "lower" don't have this).

We're trying to come up with names that we hope will trigger the correct associations in the API users' brains, but there seems to always be a nontrivial chance it will trigger an incorrect association as well, and that makes it strictly worse than the medical names because not only do they have to look at a diagram anyway to work with these joints, their brain will actively confuse them since it has latched on to an unintentional mnemonic.

By coming up with our own names for something people do not otherwise have names for, we are not doing away with jargon, we are inventing our own jargon that just sounds less jargon-y.

A crucial part of that TAG guideline is "Whenever possible": I don't think this is a situation where that sort of naming is possible without ambiguity.

[cabanier]

maybe we can keep the names for the finger tips and then start counting

I already spoke out against this in the issue: the problem is that not every hand tracking system exposes the same joints, so how you count is tricky. Especially around the thumb, where the choice of joints is rather variable.

We already agreed that each implementation HAS to expose the same joints.
Maybe we can enhance this in a later revision of the spec but for now, the UA has to expose the 25 predefined joints and either emulate missing ones or skip the ones that aren't in the spec.

Furthermore, the direction of the numbers can be confusing, and folks will have to remember if it's 0-indexed or 1-indexed. This does not seem to be solving the problem, people will still have to refer to a diagram.

If you "start" with tip, there is little chance of confusion. (I suspect that the majority of experiences will only use finger tips.)
People never start counting from 0 so beginning with 1 makes the most sense.

So I'm reluctant to go back to names that people will have to look up on wikipedia.

My contention is here, no matter what we pick, they will have to look up a diagram on some docs (MDN, etc), so the best we can do is pick something without ambiguity.

tip and then 1,2,3,4 is not ambiguous and easy to remember. I still can't remember the medical terms and implemented the feature and wrote several examples.

...
By coming up with our own names for something people do not otherwise have names for, we are not doing away with jargon, we are inventing our own jargon that just sounds less jargon-y.

People know what finger tips are and by adding numbers, we're not inventing jargon.

A crucial part of that TAG guideline is "Whenever possible": I don't think this is a situation where that sort of naming is possible without ambiguity.

I'm not ready to give up just yet :-P

[Manishearth]

We already agreed that each implementation HAS to expose the same joints.

No, I mean other APIs, not just WebXR. Basically, other APIs trying to number joints will do things differently and it'll be confusing.

tip and then 1,2,3,4 is not ambiguous and easy to remember. I still can't remember the medical terms and implemented the feature and wrote several examples.

I don't think so, because my immediate read of tip + 1234 is that the 1 starts at the bottom -- because that's how I would naturally number the fingers, and then it just feels weird that the numbering is "1 2 3 4 tip", even though that's not actually what it is.

People know what finger tips are and by adding numbers, we're not inventing jargon.

Yes, but the numbers could start at tip or could start at the bottom. We are inventing a new scheme for people to understand and remember, except that it's a scheme where intuition can apply in the wrong direction, confusing people. On the other hand, people will not have intuition about the medical terms.

[cabanier]

We already agreed that each implementation HAS to expose the same joints.

No, I mean other APIs, not just WebXR. Basically, other APIs trying to number joints will do things differently and it'll be confusing.

I don't understand what you mean by "other APIs". Are there plans to expose joints to non-WebXR content? If so, would they offer more joints?

tip and then 1,2,3,4 is not ambiguous and easy to remember. I still can't remember the medical terms and implemented the feature and wrote several examples.

I don't think so, because my immediate read of tip + 1234 is that the 1 starts at the bottom -- because that's how I would naturally number the fingers, and then it just feels weird that the numbering is "1 2 3 4 tip", even though that's not actually what it is.

Starting at the tip and then counting feels very intuitive. If you hear it once, you will not get confused again or have to look up a technical term.

People know what finger tips are and by adding numbers, we're not inventing jargon.

Yes, but the numbers could start at tip or could start at the bottom. We are inventing a new scheme for people to understand and remember, except that it's a scheme where intuition can apply in the wrong direction, confusing people. On the other hand, people will not have intuition about the medical terms.

Yes and that is a problem. It has to be understandable. Even looking at hands code people will have to look up the technical terms.
So with numbering, some people might be confused but with the medical terms they certainly will be.

[Manishearth]

I don't understand what you mean by "other APIs". Are there plans to expose joints to non-WebXR content? If so, would they offer more joints?

The oculus API also numbers things. People used to one API will have expectations from that API confuse their interpretation of another API.

Starting at the tip and then counting feels very intuitive. If you hear it once, you will not get confused again or have to look up a technical term.

It isn't for me, partially because of the time I've spent looking at the Oculus API, but also partially because I have a strong expectation the numbers will go up as you go towards the tip.

Like, when I saw the suggestion I immediately made the wrong interpretation and then thought more and realized what you meant.

Yes and that is a problem. It has to be understandable. Even looking at hands code people will have to look up the technical terms.
So with numbering, some people might be confused but with the medical terms they certainly will be.

With the medical terms there is no intuition to trip you over. With the medical ones people will have a hard time remembering but they will not be confused -- there will not be situations where they feel reasonably certain their interpretation is correct when it isn't.

[cabanier]

I don't understand what you mean by "other APIs". Are there plans to expose joints to non-WebXR content? If so, would they offer more joints?

The oculus API also numbers things. People used to one API will have expectations from that API confuse their interpretation of another API.

I'm unsure if I buy that argument. Generally, the web platform concerns itself with APIs and API names that make sense for an author.
By your argument, maybe we should stick with my original proposal because it's very close to Magic Leap's hand tracking API.

Starting at the tip and then counting feels very intuitive. If you hear it once, you will not get confused again or have to look up a technical term.

It isn't for me, partially because of the time I've spent looking at the Oculus API, but also partially because I have a strong expectation the numbers will go up as you go towards the tip.

Like, when I saw the suggestion I immediately made the wrong interpretation and then thought more and realized what you meant.

Sure, but will you ever make that mistake again?

Yes and that is a problem. It has to be understandable. Even looking at hands code people will have to look up the technical terms.
So with numbering, some people might be confused but with the medical terms they certainly will be.

With the medical terms there is no intuition to trip you over. With the medical ones people will have a hard time remembering but they will not be confused -- there will not be situations where they feel reasonably certain their interpretation is correct when it isn't.

They are almost impossible to remember which violates the TAG guidelines. @alice, please chime in if you also feel strongly about this.

It seems that we're at a standstill so maybe we should put it to a vote in the next meeting.

[kdashg]

Guidelines are rules that bend when there is good reason to bend them.

[Manishearth]

By your argument, maybe we should stick with my original proposal because it's very close to Magic Leap's hand tracking API.

No, that's not the logic I'm applying. I'm saying that if we use numbers and there's a different API using numbers, users familiar with the other API will get confused. This does not mean we should copy a particular API for familiarity.

Sure, but will you ever make that mistake again?

Depends on how often I'd be writing this. I feel like after not dealing with gesture code in a while I would probably come back and make the same mistake. It's not like people will be constantly using these names, they'll use them once and then not for a while until they come back to this code. In the meantime they may have used other APIs with other numbering schemes.

They are almost impossible to remember which violates the TAG guidelines. @alice, please chime in if you also feel strongly about this.

The TAG guidelines say "when possible" 😄, but yes, it would be useful to have feedback from the TAG on this.

[Manishearth]

Worth noting: OpenXR, another standardization body, already went through this process and decided to use the medical names for roughly the same reasons as @thetuvix was saying. They don't have the TAG guidelines but it does seem like this was explicitly considered.

[alice]

Yeah, we came to the same conclusion in TAG as well - it would be ideal if there were some other non-ambiguous vocabulary to use which was easier to understand, but given there isn't, the medical terms are the best available choice.

[cabanier]

ok. It seems that most people are ok with the hard-to-remember but correct terms. I will update the PR.

[cabanier]

@Manishearth @domenic Can you review my PR?
I'm a bit worried about the long enum names. Should I remove the -finger part?

[Manishearth]

I don't have a strong opinion either way, but curious as to what others think!

[cabanier]

I don't have a strong opinion either way, but curious as to what others think!

Should we put it on the agenda again?
I would like to have this in soon so I can submit my browser changes and people can start updating their code.

[Manishearth]

Should we put it on the agenda again?

We could. Wondering if @alice or @thetuvix have opinions.

[cabanier]

/agenda any comments on the Hands redesign?

[cabanier]

I would like to get this in as soon as possible because a lot of content is being created for the old API...

[domenic]

API design looks good; just some comments on spec formalisms.

[cabanier]

closes #70

[cabanier]

@Manishearth I ran this by 3 different framework developers and it was discussed during this week's WebXR call.
I think we're good to merge this.

[Manishearth]

Sweet! I'll have a look soon, we should also update the explainer

[Manishearth]

Thanks for all the discussion, everyone!

[fordacious]

@Manishearth @cabanier Just an aside. Do you have an estimate of what the perf impact for this change would be? Are there any differences moving to a maplike structure?

[Manishearth]

I don't think so because the internals can still be implemented as an ordered array. It's implementation dependent (happy to look at any browser implementation changes if you need me to!)

[cabanier]

@Manishearth @cabanier Just an aside. Do you have an estimate of what the perf impact for this change would be? Are there any differences moving to a maplike structure?

If the author was looking up each joint for each frame by name, there might be some regression.
However, pretty much every case just iterates over the hand so there should be no change in performance (ie my patch for three.js)

Also, the list of spaces is defined to be always the same so an author could just cache them if they wanted.

[Manishearth]

Yep, in the case where you're doing specific joints (e.g. for gestures), you can use the same joint space each time, there is no need to fetch it over and over again.