Use 'data-at-shortcutkeys' to pass access keys to JAWS

[jessegreenberg]

From #109:
@terracoda said:

I think we should try the JAWS data-at-shortcutkeys technique, on the Play Area section tag? It would be nice to see how it works with JAWS at least. It might look like this:

<section data-at-shortcutkeys = { ‘?’: ‘Key to open command shortcuts',‘jw’: ‘Keys to jump to Wall’, ‘jn’: ‘Keys to jump to Near Wall'}>

I do not know how to represent an Arrow Key in the dictionary pair above.
Some people posted that these work arounds are better than a developer using "role=application".

@jessegreenberg said:

We will have to see how JAWS presents this information, if at all. I expect that the key value pairs will be read so hopefully we can describe the arrow keys in any way we wish. But perhaps it literally maps the key to be passed in? That would require key codes for things like the arrows, but I haven't found an example of that yet. I am going to test all of this now.

Here are some results after playing around with this.

[jessegreenberg]

Information from Freedom Scientific here:
http://www.freedomscientific.com/training/surfs-up/BypassNavigationQuickKeys.htm

[jessegreenberg]

Testing their example, I notice that IE and FF both work. Chrome does not seem to support this.

[jessegreenberg]

The data-at-shortkeys attribute requires an object literal to be passed in with key value pairs, but the object cannot built outside of setAttribute because setAttribute seems to implicitly call setString on the object. So the data needs to be passed in to setAttribute directly in a long string. It looks like this:

domElement.setAttribute( 'data-at-shortcutkeys', "{'j': 'Key to start jumping', 'w': 'Key to jump to wall', 's': 'Key to jump to sweater'}"  );

This seems to work very well for Firefox and JAWS! It is not working yet with IE. This is the first time I am testing JAWS, but nothing seems to work with IE + JAWS.

[jessegreenberg]

In Firefox, data-at-shortcutkeys allows any key to be passed to the browser, even those that are not defined in the data object. Except for control, shift, and the arrow keys. At this time, I do not think it is possible for JAWS to pass these keys to the browser.

@terracoda provided another helpful piece of documentation: http://tink.uk/time-to-revisit-accesskey/

According to this source, data-at-shortcutkeys is something of a dirty hack because the screen reader has to parse the actual HTML. This is prone to error and less efficient than the accessibility API.

What we need is a way for websites to provide keyboard shortcuts that are exposed in the browser's accessibility tree - so that the information can be queried by any assistive technology using the accessibility API.

Until this is done, complex drag and drop with the arrow keys will not work in JAWS.

[terracoda]

@jessegreenberg said:

This seems to work very well for Firefox and JAWS! It is not working yet with IE. This is the first time I am testing JAWS, but nothing seems to work with IE + JAWS.

Interesting. JAWS is supposed to work best with IE. Btw, what version of JAWS and IE are you using?

[terracoda]

@jessegreenberg as I mentioned earlier I spoke with David Best, an expert screen reader user and programmer. He seemed to think that it would be possible to put an id on a containing element (the Play Area, for example) and use an event listener to listen for the arrow keys and then with Javascript make the Arrow keys do something else in that context. Is something like this possible? Should we discuss tomorrow?

[jessegreenberg]

Interesting. JAWS is supposed to work best with IE. Btw, what version of JAWS and IE are you using?

That was my understanding as well, so I am surprised too. I am using JAWS 17 with IE 11.

[jessegreenberg]

Regarding

He seemed to think that it would be possible to put an id on a containing element (the Play Area, for example) and use an event listener to listen for the arrow keys and then with Javascript make the Arrow keys do something else in that context.

It would be great to discuss this further in case we are missing something. That description sounds like what we are doing. Here is an example of the JavaScript:

        domElement.addEventListener( 'keydown', function( event ) {
            // functions in here define behavior for arrow keys
        } );

The event listener is added to the element. The issue is that the AT grabs the event for itself without passing to to the DOM.

Adding the event listener to a containing element would mean that using arrow keys could drag the element when it does not have focus.

[jessegreenberg]

Until this is done, complex drag and drop with the arrow keys will not work in JAWS.

I should specify that using the 'insert + 3' method will still work, but arrow keys will be intercepted for the aria roles that I am aware of.

[terracoda]

@jessegreenberg regarding event listener and arrow keys, David mentioned passing "Insert+3" as an example (keep in mind this is a JAWS-only hack) in that context to prefix the key event. Is something like that possible?
If focus on the balloon, all keystrokes are prefixed with the pass through.

Some how I thought this would already happen naturally with the input element. It would be great for you to meet David.

[jessegreenberg]

Yes that is possible, that is what is happening now. By pressing 'insert + 3' the event is passed to the browser, no matter what the element is.

Input elements naturally get certain keystrokes depending on the type. For example, buttons get spacebar and enter. All other keystrokes are intercepted.

[terracoda]

Can the application send 'insert+3' for the user? Is that what you are describing?

[jessegreenberg]

That is not what I was describing, I hadn't thought of that! Perhaps that would be possible, maybe with something like dispatchEvent? It feels a little hacky, but I will need to play with it and see if this could work. I am not sure if we can dispatch events from the browser to the screen reader.

[terracoda]

It is very hacky, but that is what David described. It would not be a long term solution. Garaventa's wise words are ringing in my ears :-)

Interactive components should never be specifically programmed to work within specific AT/browser combinations, such as using ARIA hacks to make something work in Firefox and JAWS, when doing so results in non-standards compliant markup and inaccessible functionality elsewhere.

I'd still like to see what David was thinking.

[jessegreenberg]

I am curious about this approach as well, looking forward to discussion at the meeting. Though I do not like the idea of forcing the device to do something that the user did not ask for.

[terracoda]

@jessegreenberg Thinking back in history of this design idea. The original idea was to grab the balloon with Enter key. Since <input type="image"> behaves like a button, the Enter key is a native interaction. maybe we should have an explicit grab interaction to activate, or "grab" the balloon. I wonder if that would help? What do you think?

[jessegreenberg]

That is an interesting idea @terracoda. I can't picture yet how we could use this to solve the issue. I believe keyboard interactions would still be intercepted after pressing the button and we can't use the button to toggle AT modes. Were you thinking of a description design like

"Enter forms mode to grab the balloon"

or something like that?

[terracoda]

Actually, @jessegreenberg, I was just thinking
"Enter to grab balloon."
I do not believe that an average screen reader user has much awareness about what mode they are in. It is even less likely that they would know the name of the mode. A user is aware they are on a form element, but not necessarily that the screen has changed from browse mode to forms mode.

[jessegreenberg]

I see. I think you are right, it should ultimately be up to us to seamlessly toggle the AT between forms and browse mode

Unfortunately, I don't think interacting with the button will help with this issue. Nothing changes for the AT once the button has been pressed in terms of interaction modes.

[terracoda]

This is frustrating. @jessegreenberg, I am grasping at straws here, but there may two HTML syntax issues with the code that might be causing issues:

1. It is not valid code for an input element to have a closing tag.
2. The parent of the input element is technically supposed to be an element that can contain Phrasing content. The parent element of the balloon input is a <div> element which is technically a Flow content element.

These details are on MDN

• https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories#Phrasing_content
• https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-type
• https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/image

We discussed the incorrect closing input tag before. As for Phrasing content a span qualifies.

No idea if this would matter, but the code would be very correct.

[terracoda]

@jessegreenberg one more thought. There is such a thing as a double slider, a slider with 2 values. Am I dreaming, or could one value represent the x-value and one represent the y-value, and still be one control? Hans Hillen discusses ARIA Sliders in a 3-part series. He has a page of examples as well.

Could values 0-99 represent the x axis and then 100-199, the y-axis? Could you restrict the the left-right arrow keys for x values and the up-down arrow keys for y-values?

[jessegreenberg]

@terracoda, these are all great points, and I think they could be causing issues for the screen reader beyond just the key press interception.

To respond to your comments:

1. I see that a few input elements have child nodes, which is not valid HTML. The closing tags must have been generated to contain this content. I will restructure the sim slightly so that the <input> and its description are contained in a block. The input should not have children. I think this may be the reason JAWS is unable to read our content in IE.

2. My understanding from the above documentation is that phrasing content is a subset of flow content. The div is a generic container for flow content, which means that it can also contain phrasing content.

3. I considered the double slider when we were discussing Determine best accessible representation of Balloon #108. We could pursue this option, but I decided against it for a couple reasons:

• The double slider is meant to be used to restrict searches within a range.
• The double slider creates two inputs from one. The user must tab to each individual thumb of the double slider to adjust it. It probably would be possible to restrict the arrow keys for the x and y directions, but the user would first have to tab to the correct thumb.
• The screen reader would still intercept shift, enter, and spacebar.

[jessegreenberg]

What if the balloon were represented by an input of type text? Continuing the conversation here:#108 (comment) since it pertains to that issue.

[jhung]

@jessegreenberg @terracoda
Justin and I have been following this discussions. We played with the demo and we can see the challenge you’re dealing with. We didn't know what would be the best solution in this case, so we’ve forwarded this issue to Joseph Scheuhammer here at the IDRC who is on the ARIA advisory board at the W3C. His initial impression was that using a text-field and button role was not the way it should be done (it's not a text field, nor is it button-like in behaviour), but he couldn’t think of an immediate solution. He’s going to do some research and consult with some colleagues, and we hope he can advise us on how we should approach this.

Do you want to wait and see what he comes back with?

[jessegreenberg]

@jhung, Thank you for reviewing this issue! I agree with all concerns about the translation. The draggable balloon is not a text field nor a button. I considered the input text to be a short term workaround. It is just a way to get us up and running until accessibility API's and screen readers let us pass all desired key events to the browser for application behavior. A text field was the only input I could think of that would accomplish this.

Any suggestions would be wonderful, we would appreciate any recommendations from Joseph Scheuhammer.

[terracoda]

@jhung thanks for this initiative. I have also discussed this problem with Dr. Clayton Lewis, my thesis advisor, and was planning on reaching out to the Web Standards community. Maybe we could coordinate this effort. Jesse and I discussed this issue a lot, but not in a direct face-to-face brain storming.

[jessegreenberg]

In Issue #116, it was determined that the application role works to specification in most browsers with the letter keys. The arrow keys are generally intercepted regardless of the role. See issue #116 for more information.

Considering this, I am inclined to replace the text input with a div for now to represent the balloon. As long as we are OK with using WASD (or other) keys for drag and drop, this should work in most browsers without exploiting the UI of the HTML text input.

[jessegreenberg]

Progress with the application role is being tracked in #116. data-at-shortcutkeys is not a route we want to go down unless absolutely necessary. It is a solution that is specific to JAWS and only works with a few browsers. Closing.

[terracoda]

Thanks for closing @jessegreenberg. Agreed.