Second thoughts on pointer gestures and drag-and-drop

[detlevhfischer]

I have some (late) second thoughts on the new SC Pointer Gestures.
Take a look at the Salesforce example of drag-and-drop that has purpose of laying out elements on a two-dimensional plane (as in composing a diagram) referenced in a recent accessible drag-and-drop article on Medium by Jesse Hausler.
This drag-and-drop example has been made keyboard-accessible (with added instructions for desktop screen reader users via aria-live) and also allows dragging with a single pointer (including touch). Now, do we really mean there should be an extra mechanism to move the objects with single clicks / tabs? This seems to necessitate either

• four discrete buttons at each side of a moveable object, or
• a mechanism (e.g. single tap) whereby the object can be selected / "picked up" and on-screen controls (arrows) are then available to move the selected element in discrete steps

I fear we might have gone too far; I fear that adding extra functionality for draggable objects to afford single click/tap operation might decrease the usability / affordance overall, and may not be worth it (quite apart from adding a lot of complexity for developers). Even so, the current SC text seems to mandate exactly that, and would fail the example above. Since accommodations are possible, this does not seem to be a case for the 'essential' exception - but others may disagree.

Thoughts?

[patrickhlauke]

I'd note that the intention for the SC is to prohibit path-based gestures. i.e. where a user has to follow a specific path / make a specific gesture (e.g. a precisely timed and executed swipe or similar). Dragging/dropping is more of a freeform interaction...while the user is dragging, it doesn't matter if they stray off a specific path or not, what counts is only the start and end points. Maybe this should be clarified (assuming my take on this is correct).

[patrickhlauke]

to further clarify: for a path-based gesture, a user has to have enough fine motor control to "down" the pointer (e.g. put the finger on the touchscreen, click and hold the mouse button, etc), follow a specific path within a specific time (as otherwise it doesn't count as, say, a swipe), and then release the pointer (e.g. lift finger off the touchscreen, release the mouse button, etc)
for a drag'n'drop operation, the user has to only "down" the pointer (to select the item), and is then free to move the pointer as slow/fast and to whatever position they want (though the system may constrain the area where the element can be dragged on). they don't need to be particularly fast nor precise. what counts then is only where they stop - the end point - where they then release the pointer.

[patrickhlauke]

having said all that, an alternative method for allowing a drag'n'drop operation that doesn't rely at all on movement would be: allow the user to tap/click an item to move; the item is then "selected" - user doesn't need to keep the pointer down (i.e. they can release the mouse button, lift their finger off the touchscreen); the user then taps/clicks on the destination they want to move the item to. so a "pick this thing, move it there" kind of paradigm, which does not require the user to keep the pointer down at all.

[detlevhfischer]

Yeah - "Select and then click /tap in the direction where the user wants to move the element" crossed my mind as well, but I think in many contexts (especially on more crowded canvases), interpreting such a tap might lead to unintended results. Someone implementing that would have to work out whether the horizontal or vertical offset (element centroid and tap/click location) is larger; would need keep the element selected and thereby negate the option to deselect by tapping/clicking outside of it, which would be bad if it moves under another element higher up on the x-axis; work out what to do when a click/tab happens to land inside another object (should this element now be immediately selected?) etc. Maybe all this is solvable, but gut feeling tells me it is probably not the right approach.

As to differentiating path based gestures like swipes from 'free-form' gestures in drag-n-drop: I would dearly like to buy into that distinction, but my hunch is that flicks and swipes, if anything, might be easier to perform than the sequence of tap on a particular target (to pick up), hold and drag (to move) and release over a defined area (to drop).

[patrickhlauke]

interpreting such a tap might lead to unintended results [...]

it would be a case of having this as a toggle-able mode. i could envisage once in the mode, after first tap to select, you could have a new bar at the top of the screen to exit mode without moving/dropping. or tapping on the same/selected element again to deselect. there are options in any case. but also, i still believe there's a distinction here between a "path-based gesture" and a drag'n'drop interaction.

as for gestures being easier than drag'n'drop interactions: note that gestures usually have a time element to them (needing to execute a specific path within a certain threshold of time, otherwise it's not registered as a gesture), and straying from the path invalidates the gesture. not all gestures are just simple horizontal/vertical swipes either. think for instance the various Android/TalkBack gestures (L shape ones like down-right etc). (and yes I know TB gestures are exempt, but just giving you an example of a gesture that, if implemented for web content, would be difficult for users to accurately perform within a set amount of time).

and i would still say, in general: a drag'n'drop interaction is, to my mind, NOT a "gesture" (gestures being essentially commands that require the user to draw a particular path within a certain time threshold). [edit: and to really boil the point down: drag'n'drop is NOT path-based, as the user does not have to follow a specific path ... so even IF we want to argue that drag'n'drop is a "gesture", it remains that it's not path-based so the SC does not apply as the normative language clearly says "path-based gestures"]

[mraccess77]

I believe the intention was that path based gestures include drag/drop and swipes as well. It's interesting that we decided that drag and drop under Pointer Cancelation had to have an undo/abort as that could help people recover from a mistake with a drag and drop gesture. I agree that dragging and dropping will be problematic for some users. We should examine how each type of user might use this and what the solution is. For example, on mobile like iOS with switch control -- how would a user perform the drag and drop without using point mode? But at the same time how would they use a mode with an infinite number of targets without having buttons to move left, right, up, down? I agree a swipe to move would help -- but as intended swipes would be path based gestures. Ultimately what is needed a the ability to bring up a list of actions and to be able to choose an action from that list. Then the actions could be buttons that are context sensitive and would work with switch control or other technology. The challenges are very different here for users who are blind versus users who rely on switch or other input modalities. While an author could bring up an actions lists and that may be needed -- ultimately having an API to communicate what the actions are and to perform those actions would be best and would move this from the author requirement to the user agent.

[patrickhlauke]

we need to also be careful here not to try and solve everything in this SC. note that the SC for keyboard is still present, and that will cover things like switch access too.

As reminder, the normative wording for this SC: "(Level A)
All functionality that uses multipoint or path-based gestures for operation can be operated with a single pointer without a path-based gesture, unless a multipoint or path-based gesture is essential."

So, this isn't about "must be able to perform this without any pointer interface" (e.g. keyboard or switch access), but rather that where a user can use a pointer, they're not required to perform path-based gestures (complex due to path-following/precision and timing) or forced to use multitouch gestures.

[mraccess77]

@patrickhlauke The point of this SC was to not require users to carry around a physical keyboard to use a mobile site and to address these situations where something may technically be keyboard accessible but otherwise very difficult or impossible to access without a difficult gestures that either couldn't be performed with their AT or difficult for them to perform without AT. As you know it can be difficult if not impossible to get a keyboard to appear in a mobile browser. If the browser supports arrow key navigation -- how would a switch user be able to send those left/right/down/up keys through switch control? As I said before ultimately these issues should be solved through an API at the user agent level allowing authors to specify actions and for users to query a list of actions. Yes we still have the keyboard requirement but the intent is to minimize when users have to fall back to that requirement. The associated understanding document talks about not relying on gestures and I think the understanding document describes then intention of the group when this was written. It sounds like we are now in a situation where we need to scale back the original intention through re-writing what we "really meant to say was". If that's what we have to do then we should acknowledge the change. This seems to be a common theme across many of the SCs.

[mraccess77]

I tried the example from Detlev with VocieOver on iOS and without a keyboard I was not able to tell where I was moving the object. With a double tap, hold, and drag I did not get feedback and VocieOver started selecting text and such. This is the type of situation that we were trying to solve with this SC.

[detlevhfischer]

Maybe we should cover this point in the understanding document. Authors will want to know if an implementation like the Jesse Hausler one I pointed to would pass or fail SC Pointer Gestures. If others, like Patrick, think that a drag-and-drop action does not fall under "path-based gesture", that problem is solved, (and I would be relieved).
BTW my impression is that the timing aspect of gestures (will it be interpreted as flicking or tapping? will it work?) is most pronounced in AT gestures (especially on Android where it can drive users to distraction), much less so in drag gestures (AT turned off) implemented on web sites (typical is the swipe-to-reveal-next-slide type carousel in news sites). Compare the weather forecast slider two-thirds down on tagesschau.de on a moble phone - here, neither speed nor exact direction matter much, the slider responds.

[patrickhlauke]

The point of this SC was to not require users to carry around a physical keyboard to use a mobile site and to address these situations where something may technically be keyboard accessible but otherwise very difficult or impossible to access without a difficult gestures that either couldn't be performed with their AT or difficult for them to perform without AT.

maybe that was the original intent, but I'm certainly not getting that impression from reading the normative SC text...

[detlevhfischer]

@mraccess77

With a double tap, hold, and drag I did not get feedback and VocieOver started selecting text and such. This is the type of situation that we were trying to solve with this SC.

Apart from adding more arrow controls (appended to the individual object, or general ones referencing the selected object) is there anything the author could have done to ensure voiceover output of position in that pass-through mode of repositioning? Beyond this example, there are ARIA widgets that do not work on iOS with VO on. Would that in your view always constitute a failure, or is is a UA/AT issue the author has no control over?

[patrickhlauke]

interesting point about swipes that are not timed. yes, i was perhaps thinking too much about gestural commands rather than...direct manipulation type gestures. interesting conundrum then in how to word this (as i believe we introduced "path-based gesture" to differentiate it from simple taps, which also fall under a generic "gesture").

on the "don't just force users to directly manipulate/swipe", i'm thinking now of the iOS mail app, where you can slowly swipe an email left/right to reveal options (archive, delete, flag, etc), but then the same actions can also be performed by simple taps (when tapping through to the email first, and then using the regular delete, flag, etc buttons at the bottom of the screen).

going back to drag'n'drop as in your example - which is essentially visually arranging elements (and focusing purely on the "can tap, just not accurately move/swipe" scenario, and leaving aside switch/keyboard/touch+AT scenarios) - i believe offering a two-step "select, then choose the target" mode would work here (with an additional "Cancel" option that shows up for users to bail out of the operation altogether, or allowing them to tap on the element again without moving it)

[mraccess77]

on the "don't just force users to directly manipulate/swipe", i'm thinking now of the iOS mail app, where you can slowly swipe an email left/right to reveal options (archive, delete, flag, etc), but then the same actions can also be performed by simple taps (when tapping through to the email first, and then using the regular delete, flag, etc buttons at the bottom of the screen).

For what it's worth -- with VoiceOver running in the Mail app the VO user can swipe up or down on the message and go through rotor settings to quickly delete, mark as unread, etc. So VO users will swipe up which is delete by default and then double tap.

[patrickhlauke]

@mraccess77 yes but that's not a type of functionality that's available to web content (populating the rotor with custom commands)...only native apps can do that.