Update pointer-gestures understanding doc

[mbgower]

Addresses resolution from group discussion for #522

I removed most (but not every) dragging example, and added in language about the desirability of alternatives.
I also made some minor editorial changes for readability which are not intended to alter in any way the meaning of the Understanding document.

[detlevhfischer]

Summary: I am afraid I am not happy with this PR. The distinction between swiping and dragging is not as clear-cut as we may wish. I would prefer to keep constrained dragging in scope, and single out free-form drag-and-drop as being outside scope.

---

In preparing this review I have looked at a number of swipe/drag implementations. Here are some observations:

(1) From a user perspective and in terms of the needs of motor-impaired users, there is no clear conceptual separation between swiping and dragging - all elements that react to swipe can equally be dragged (all swipey things I tested) though not all draggable things (thump of slider) can be swiped (possibly due to their limited footprint).

(2) A distinction that I initially thought valuable, that swiping can start anywhere within the object while dragging has to pick up the object, may be spurious: the object that is swiped (or dragged), like an image slider, is just a lot bigger.

(3) The characteristic that @mbgower notes for some objects that they can be

dragged in a wayward manner to its destination

is only true for some draggable / swipable elements - in a mobile browser context where vertical swiping/dragging will usually be interpreted as vertical scrolling of the page, there will need to be an initial horizontal movement before the x position of the element will be set to the x position of the pointer regardless of the pointer's y position - compare smartslider 3 or ARD Wissen. It is equally possible (I am lacking examples for that though) that the dragged object's x position would only follow pointer x within bounds, for example, only as long as the pointer moves over a rectangle containing the slider control, which would mean that the gesture would have to be (roughly) directional or fail. Finally, the noted wayward characteristic can be true for control slider thumbs as much as for the type of swipable/ draggable horizontal content slider that remains an example in this PR (example 2).

(4) There are complex swipe / drag gestures where the functionality triggered can be dependent on the length of the path (along the dimension that is not constrained) - compare an Oracle implementation of swipe-to-reveal. A long path triggers deletion, a shorter one reveals options. The behaviour of the control is independent of whether users perform a quick swipe gesture or a slow drag gesture -- it is the length of the path that is decisive.

@mbgower has argued that there is no essential difference between dragging a slider thumb and a free-form drag and drop scenario. I would argue that the affordance of constrained dragging in one dimension is very different from free-form drag-n-drop. In the former, the element that moves stays visibly locked to the constrained dimension and does not follow the pointer around. I think the difference is sufficient to draw a clear line if we agree to keep drag-and-drop out of scope for reasons of inplementation complexity.

I would suggest to add an example of the swipe-to-reveal type as complex path gesture requiring an alternative (Oracle or React swipe-to-reveal)

I gather that we may prepare an additional SC for 2.2 to cover draggable stuff separately, but I see no need for that if we can keep swiping and dragging covered here - the normative text seems to allow that. WCAG is getting very bitty, and I would prefer treating closely related aspects in one SC.

This is a tentative definition of "path-based gesture" that includes both swiping and dragging:

"A path-based gesture triggers some function when users, after initially engaging a pointer with the display (down event), carry out a directional movement before disengaging the pointer (up event). The direction, speed, and also the delta between start and end point may all be evaluated to determine what function is triggered.
Typical functions triggered by path-based gestures are changing the value of a slider, advancing an image slider / carousel, revealing interaction options attached to an element, or opening or closing a menu drawer or other popover content. Free-form drag-and-drop interactions where the direction and speed of the gesture are not evaluated for triggering a function are not considered path-based gestures."

[patrickhlauke]

After looking at this from different angles, I'll admit that I'm torn on this issue myself. There are certainly bad/naive implementations of sliders that do not offer all the good affordances such as allowing users to stray off the slider track, and essentially require users to very precisely follow the slider's track/path very closely. I would count these (bad) implementations as failures under this SC unless they provided alternatives (such as the ability to tap/click anywhere on the track to jump the slider button directly to that position/value, or extra arrows/controls to move the slider up/down or left/right). so discounting/out-of-scoping sliders altogether here feels wrong to me.

A rather broken example of such a bad mouse-driven slider can be found here https://codepen.io/patrickhlauke/pen/yMpbXa (and yes, it has many other issues, such as not being AT or kbd friendly, but just to demonstrate the problem of slider thumbs not "capturing"/"locking onto" the user's pointer when clicked) - compare to this other example (as broken as the previous one, but at least it follows the mouse's X position even after the user moves the mouse outside of the slider) https://codepen.io/patrickhlauke/pen/eZgqRo

[edit:] for clarification then, my quickfire view (focusing purely on the 2.5.1 aspect here of path-based gesture) would be that the first non-capturing example fails, while the second (while still problematic for many other reasons) passes (barely)

[detlevhfischer]

@patrickhlauke I think the issue of Mike's proposed edit is if sliders would be considered out of scope, which would exempt a significant group of interface objects from being operable by single point activation. The addition of best practice recommendations

Authors are encouraged to provide non-dragging methods

cannot compensate for that removal. And I do not see a clear way of differentiating between control sliders (then out of scope) and content sliders (still in, if judging by the examples) so authors may assume any slider is exempt. By contrast, exempting just free-form drag-n-drop (the object just follows the pointer without being constrained in any way) seems a clear-cut exception if the issue really is "explaining that to the developers".

Whether the slider implementations you point to would be in scope due to the path being or not being constrained vertically seems a different point. From the perspective of single-point activation (click or tap at one point) they both fail since you can't click / tap the groove to reposition the thumb (and there are no alternative inputs for incrementing/decrementing).

[patrickhlauke]

as @detlevhfischer asked me to clarify (since my usual word salad may have buried the lede): when I said "I would count these (bad) implementations as failures under this SC" I meant, in a roundabout way "I would not remove/exempt sliders from this SC outright, as there are slider implementations that can be problematic for users in ways that this SC aims to actually cover".

or shorter: no, i wouldn't exempt sliders.

[alastc]

It's a good point about swipe-to-reveal, it is somewhere between a slider and a gesture.

Can we draw the line just prior to full drag and drop? I.e. drag 'n drop has specific start and end points, and the line between them does not matter.

Whereas a slider / gesture does not have a specific end point.

I still think the rational that drag 'n drop is too complex (with all the other criteria to account for as well) is the best one for this purpose.

I've done a PR (#725) to take Mike's readability updates, keep sliders, and add an example for short/long swipe.

Feel free to update that, I'll pop this whole question back in the survey for next week, with two clear options.

[jake-abma]

just some 2 cents, we've implemented sliders using drag-and-drop from a technical perspective...

[mbgower]

From a user perspective and in terms of the needs of motor-impaired users, there is no clear conceptual separation between swiping and dragging

Gotta disagree with this to some degree. One swipes in a direction. Therefore, it's directional and so must be path-based. The simplest example I can find is in the Yahoo mail app. I go to an item and put my finger on it. Each direction I pull causes a different action. (up/down, scroll inbox; right, delete message; left, mark as read). I can't turn one action into another; it's triggered by direction.

That's clearly path-based, to me.

I'm happy to see you incorporate that into a proposed new rewrite of Understanding and get traction on that once there is a consistent, easily understood distinction.

There are complex swipe / drag gestures where the functionality triggered can be dependent on the length of the path

If it's a complex swipe/drag that has dependencies, then obviously it wouldn't qualify in the exception. Add in language about that in the doc.
I don't think timing is called out in the document. Since we don't define 'complex' I think we have the ability to introduce timing into the discusion in the Understanding document. To me, if the speed of gesture alters it's interpretation (I guess we'll need limits for that) then it is a clear indication of a complex gesture.

[patrickhlauke]

perhaps muddying the waters further, "swipe" at least to me also suggest more of a fast motion, while dragging is more measured/slow. though you then also get interfaces/components where you can either slowly drag or "throw" things (with momentum etc).

wondering if the distinction in general is more about "you can do things by doing simple taps/clicks, without the need to move a finger on the screen/keep mouse pressed and move it" (though our original intent was, i think, that movement itself was fine, just not the need for precise/following a particular path type movement...but maybe that isn't useful enough to disambiguate situations?)

[detlevhfischer]

The question has been raised before - and more recently by @mbgower - if the definition of "path-based" that we intend to add needs to be in line with the exception for path-dependent input in SC 2.1.1 Keyboard:

[Everything is keyboard-operable] "...except where the underlying function requires input that depends on the path of the user's movement and not just the endpoints."

"Note 1: This exception relates to the underlying function, not the input technique. For example, if using handwriting to enter text, the input technique (handwriting) requires path-dependent input but the underlying function (text input) does not."

The path-dependent definition in 2.1.1 was, as I see it, motivated by functions (drawing, signing) that require the production of a particular shape - something that cannot be achieved with a keyboard. ‚Path’ is thus a component of such a shape. Its nature is not input-operational but descriptive (for want of a better term) - what counts is the resulting form that may be taken as-is or pattern-matched / converted.

In contrast, the purpose of the pointer input paths we are dealing with here is operational - moving something represented on screen, whether content slider, menu, or control slider thumb. That is why I feel it is appropriate to depart from the 2.1.1 exclusion wording for path-dependent. I consider it reasonable (and importantly, addressing demonstrated user needs) if we now come to define ‘path-based’ as separate from 2.1.1, as operational pointer path, which would cover both swiping and dragging.

If memory serves, Mobily A11y TF's core idea of what became SC2.5.1 Pointer Gestures has been: Offer an single point operable alternative to any pointer path on screen -- for example, to enable users who simply CANNOT swipe or drag, e.g. who position the mouse with a voice-operated grid in speech AT and depend on single point activation. With that logic, free-form drag-n-drop should really be in, but I accept the argument that this would set the bar for developers very high and introduce complexities/issues for other users. This is clearly not the case for elements where movement is constrained - adding single point operation can be as trivial as making a slider groove respond to touch/click.

[detlevhfischer]

I did not intend to work on this branch and rename it - I hope that can be reverted. I have created a separate PR #735 for a proposed rewording of the understanding text that would clarify the original Mobile a11y TF intent to require a single point activation alternative for all pointer input actions that require users to draw a path (be it swipe or drag).

[jake-abma]

@patrickhlauke wrote:

though our original intent was, i think, that movement itself was fine, just not the need for precise/following a particular path type movement...but maybe that isn't useful enough to disambiguate situations?

Is that so?

When I read the part of the SC:

can be operated with a single pointer without a path-based gesture

I thought exactly the opposite.

So included or not?

[alastc]

Ach, just found this comment not posted... will discuss on the call in a moment anyway.

Hi @detlevhfischer,

You wrote (to MAFT):

I believe the original intent of the SC that became 2.5.1 Pointer Gestures was to require a single point operable alternative for all path-based interactions, including dragging. My feeling is that the issue is important because excluding control sliders from 2.5.1 via the understanding doc would affect many users with severe motor impairments.

Presumably that was adjusted because looking at drag&drop made that infeasible, and now we've zoomed in a level into the Mandelbrot picture, and are noticing different types of sliders & components.

Given the separate 2.1.1 SC, I just want to check the motor-impairments aspect, as I'm not convinced that a well-done slider should be in scope on that basis.

For example, if we took the first color-picker slider I found on the web, and made the 'thumbs' larger, I think sliding would be more effective than tapping. Being able to hold it down, experiment, get it right, then let go seems like a better interaction.

In usability testing with someone with quite a severe motor impairment they general tapped (with their wrist-knuckle), but had to use a joystick for certain interactions (including typing). In that scenario they'd be using a keyboard-like interface. For someone with a less severe impairment I think the un-restricted slider would be easier. It's hard to argue that tapping a tiny slider-target is harder than tapping a thumb-target and then being able to move freely. Unless there's something I'm missing?

Also, there are other types of widgets that fall into this little in-between space, such as resizable boxes and sort-able lists. It is difficult to see how that could be adjusted to a tap-only interface.

I do agree that using the path-based concept from 2.1.1 isn't quite the same. Perhaps we call this one 'path based gestures' instead?

[mbgower]

@detlevhfischer was quoted as saying to the MATF:

I believe the original intent of the SC that became 2.5.1 Pointer Gestures was to require a single point operable alternative for all path-based interactions, including dragging.

I was a bit surprised to see this since I clearly showed in an email to Detlev over a week ago that:

1. while swiping is mentioned in the original description, "drag" was never in the original wording provided by the MATF
2. while "path" is mentioned once, the phrase is "perform specific gestures (e.g. draw a complex path with their finger on a touchscreen) in a precise and timely manner". That seems a lot more restrictive than Detlev posits.
3. dragging as an in-scope concept is never mentioned in any meeting or issue I could find until it was introduced a year later by Detlev.
4. Greg V and others have questioned the wisdom of extending the scope in the way being advocated
5. there was a voted-on official response to an issue raised against Pointer Gestures clearly stating that path-based, as used in keyboard, was the basis of the SC wording (and that it would be turned into a normative definition, which didn't happen).

Obviously the SC was further altered after this point, but I think it's crucial we understand that this discussion has been visited and voted on multiple times (including just a few weeks ago, which triggered this PR to align the text with what was agreed on). I'm somewhat frustrated both that there are multiple discussions going on in many forums and that valid points which are raised in one place are ignored in the other discussions (like this one).

BTW, for the record I have never advocated sliders being out of scope. If someone creates a slider that depends for operation on path of movement, multitouch or (if we ever bring it in scope) timing variations, that would fail the SC. I objected to the example of dragging a slider being in scope for what I hope are now obvious reasons to do with the fact the path of the user's movement in no way affects a 'standard' slider. All that matters is the end points. Pretty much the exact same distinction used in keyboard.

[alastc]

Comments generally integrated, but I had over-hauled the middle section in parallel, so some didn't apply any more.

I haven't made any significant changes to the examples/benefits, I don't think it needed it.

[alastc]

Hi @mraccess77,

You asked in another issue something I'd rather answer here:

in your update you say a swipe gesture is a path based gesture (which I agree with) -- but I'm trying to understand how it matches the intermediate point criteria. In some samples I have seen on the Internet people are simply looking for a change in direction from the touch start location to the last touch move location (which is calculated on touch end). Maybe the platform creators do this differently than hacky devs -- but it feels like swipes are really just about the offsets of start and ending points with some tolerance for deviance of 30-60 pixels. So swipes aren't really about the start and end points but the delta within tolerances.

I think that still fits the definition, if you have to swipe in a particular direction (e.g. left), going far enough in that direction to be recognized as a swipe: So that is the mid-point.

You can carry on with the movement, and some might require you stay within a particular path, but many don't. In fact, you can often (even in some hacky web ones I've tried), go back to the start and cancel the gesture, if the X-coordinate hasn't changed enough.

However, the fact you have to go in a direction means there is a 'mid-point', the alternative would be that you could go in a different direction and still trigger it (e.g. swipe up).

[detlevhfischer]

"the relative direction of the gesture at some intermediate point" sounds rather vague and possibly misleading. The direction is critical at the onset of the gesture, to get it 'off the ground', as it were. That is also the trouble I have with the 'midpoint' language. it is really about the initiation of the gesture, is there directionality needed straight after engaging the pointer? If so, the definition of path-based is met, in my view. 'Midpoint' will probably be understood as 'half-way-through', which is the incorrect concept for arriving at an understanding of the critical difference.

[detlevhfischer]

Ah, @alastc you committed to a change while I was fiddling with another wording. This is what I tried:

"A path-based gesture involves an interaction where not just the endpoints matter - its direction (at least initially, after beginning the gesture) also affects its meaning. The user engages a pointer (starting point), carries out a movement that goes through at least one mid-point before disengaging the pointer (end point). The mid-point defines the gesture as requiring a specific path, even if the complete path is not defined.

[detlevhfischer]

@alastc I think your wording now also works, but it feels quite redundant.

[alastc]

Hi Detlev,

We tried the 'direction' approach previously (that was my thought as well), and ran into trouble. I'd rather not re-run the dicsussion again, so I'll merge it when the build passes.

[detlevhfischer]

@alastc OK, I am not insisting. Maybe I was not part of that discussion - at any rate, I do not remember why anyone objected to direction / directionality (that may just be my bad memory). But: If we aim to make our understanding texts easy to understand, it seems a no-brainer to use a more straightforward wording so people won't scratch their heads. I would imagine anyone at EOWG agreeing to that.

[alastc]

It wasn't so much objecting to the concept, as when you work the wording through it didn't include everything we wanted to due to the various implementations of swiping. I can't now remember the exact details, but from the matrix spreadsheet and the testing of different implementations, talking about an initial direction didn't cover everything.

If someone can say the same thing more simply, great, I struggled. But let's update this one now to be correct (as per the discussions) and apply another pass for improving readability later.

NB: Pretty much every addition for clarity I have made, including those from other people, has caused other people to say that change is not clear, so there is a lot of subjectivity going on!