Investigate problems role="application"

[jessegreenberg]

We need to investigate the aria application role. Our understanding is that this role should allow all key events to be passed to the web application. This has not been our experience. The AT recognizes and announces the role, but does nothing with that information and continues to use 'a 'document' mode.

It is unclear is we are using the role incorrectly, or if something about the Parallel DOM context is preventing the role from being recognized. The application role has been out for a while so it seems unlikely that it is broken.

We may have alternate solutions for this simulation, but it would be good to identify why this role isn't working as we will inevitably use it in the future.

[jessegreenberg]

From W3: https://www.w3.org/TR/wai-aria/roles#application

When the user navigates an element assigned the role of application, assistive technologies that typically intercept standard keyboard events SHOULD switch to an application browsing mode, and pass keyboard events through to the web application. The intent is to hint to certain assistive technologies to switch from normal browsing mode into a mode more appropriate for interacting with a web application; some user agents have a browse navigation mode where keys, such as up and down arrows, are used to browse the document, and this native behavior prevents the use of these keys by a web application.

[jessegreenberg]

The following is an example of accessible drag and drop from Open Ajax:
http://oaa-accessibility.org/example/17/

This example uses the application role. But the game also uses a grid role. My understanding from W3 is that this role on its own allows for arrow key navigation:

Grids allow the user to move focus between cells using two dimensional navigation.

I would like to test the above axample with a screen reader and with the application role removed. My guess is that it should work as it does now, without the announcement of 'application' from the screen reader.

[jessegreenberg]

The next test would be to listen for events with the application role in a Prototype HTML context, removed from scenery.

[jessegreenberg]

Here is one example of the application role used for the balloon in balloons-and-static-electricity. The following is the representation in the parallel DOM.

<div id="balloon-container-14-35-37-270-344" aria-live="polite" role="application" aria-labelledby="balloon-label-344" aria-describedby="balloon-description-344" >
    <div tabindex="0" id="balloon-14-35-37-270-344-345-348" draggable="true" class="Balloon" aria-labelledby="balloon-label-344" aria-describedby="balloon-description-344"></div>
    <h3 id="balloon-label-344">‪Yellow Balloon‬</h3>
    <p id="balloon-description-344">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Select WASD key to grab the balloon and start dragging. Select Tab for next item. Select question mark for keyboard help.‬</p>
</div>

[jessegreenberg]

I am confused by the example from Open Ajax. When removing the application role from the widget, the event goes on to the browser, but the X's and O's do not move to the board.

[jessegreenberg]

I just gave a test of the application role with the WASD keys. With the WASD keys, the application role works surprisingly well on most browsers! Here are my results.

For this test, I used the following html to represent the balloon:

<div id="balloon-container-14-39-42-275-349" aria-live="polite" role="application" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349" data-at-shortcutkeys="{'j': 'Key to start jumping', 'w': 'Key to jump to wall', 's': 'Key to jump to sweater'}">
    <div tabindex="0" id="balloon-14-39-42-275-349-350-353" draggable="true" class="Balloon" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349"></div>
    <h3 id="balloon-label-349">‪Yellow Balloon‬</h3>\
    <p id="balloon-description-349">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

Here are my testing results using the WASD keys for drag and drop behavior:

• NVDA + Chrome:
  • Letter keys do not work, application role ignored
• NVDA + Firefox:
  • Letter keys and application role work correctly
• NVDA + Edge:
  • Nothing seems to work...
• NVDA + IE11:
  • Letter keys and application role work correctly
• JAWS + Chrome:
  • Letter keys and application role work correctly
• JAWS + Firefox:
  • Letter keys and application role work correctly
• JAWS + Edge:
  • Letter keys and application role work correctly
• JAWS + IE11:
  • Letter keys and application role work correctly

Other than Chrome and Edge with NVDA, the application role seems to work for the WASD keys!

[jessegreenberg]

I wanted to double check the application role with the arrow keys. The following test uses the same HTML as above, but with the arrow keys replacing the WASD keys in the event listener.

• NVDA + Chrome:
  • Arrow keys do not work with the application role.
• NVDA + Firefox:
  • Arrow keys do not work with the application role.
• NVDA +Edge:
  • Arrow keys work with the application role.
• NVDA + IE11:
  • Arrow keys work with the application role.
• JAWS + Chrome:
  • Arrow keys do not work with the application role
• JAWS + Firefox:
  • Arrow keys do not work with the application role
• JAWS + Edge:
  • Arrow keys do not work with the application role
• JAWS + IE11:
  • Arrow keys do not work with the application role

[jessegreenberg]

So it looks like our main issue is with the arrow keys!

Back to the quote from W3:

some user agents have a browse navigation mode where keys, such as up and down arrows, are used to browse the document, and this native behavior prevents the use of these keys by a web application.

Looks like we can (generally) get the behavior we want for accessible drag and drop as long as we avoid the arrow keys.

[jessegreenberg]

With this information, I am inclined to avoid the text input hack suggested in issue #108. For now, I will leave the balloon represented as a div. We need to determine the best role for the balloon so that it is not announced as a section. For now, I will use a button role until further discussion.

[jobara]

@jessegreenberg i wonder if you'd consider supporting both the arrow keys and the letter keys. In that way the non-screenreader keyboard user could still use the arrow keys, and the screenreader user would also have a working keyboard solution.

We did that for our reordered component. http://build.fluidproject.org/infusion/demos/reorderer/gridReorderer/

Here you can use the arrow keys or the I,J,K,M keys. Don't worry so much about which letter keys we chose. It had to do with keys that weren't already used at the time, but it's fully configurable for an implementor to change.

[jessegreenberg]

That is a great suggestion @jobara, this is something we can do. I discussed this with @emily-phet and @terracoda and there was some concern that screen reader users and non-screen reader users would have a different experience, especially in collaboration. If I understood correctly, there should eventually be a visual cue to non-screen reader users that the WASD keys can be used to drag the balloon.

[jessegreenberg]

That being said, perhaps we should discuss again. I don't particularly see any harm in enabling both arrow and WASD keys for convenience to non - screen reader users.

[jessegreenberg]

That is great example @jobara, thank you for sharing! Having both methods of interaction available makes it robust and convenient.

With the application role, sometimes arrow keys work just enough with the screen reader to pick up the balloon, but not to drag it. It looks like in some cases the keyup event is fired immediately after keydown for the arrow keys. Enabling both in this case could make the simulation seem broken when arrow keys are not fully supported.

I should note that arrow keys and WASD keys always work perfectly if the user forces the screen reader into 'forms' mode (or whatever that mode is for the AT).

[terracoda]

@jessegreenberg, I agree with @jobara that supporting both letter keys and arrows is a good idea. I haven't given up on arrow keys and only thought the WASD keys were a temporary work around while we figured out how to best represent the balloon in code. I certainly see the balloon as a drag and drop, always have. I was more concerned about complicated interactions with the kinds of examples I seen.

The example above is a much simpler drag and drop interaction than other drag and drops I've seen. It's actually the kind of interaction I was envisioning when I brought up the menu role idea as a possible solution to get the arrow keys to work.

And I also agreed with @jobara that the menu role was not ideal, and that a deeper look at how to make a simple drag and drop was preferred. Thank you @jobara for sharing. I didn't realize you had already built a simple drag and drop.

In the case of the balloons, I feel one needs an explicit release (or drop) because the balloon doesn't necessarily stay where it is dropped. @jobara is the explicit drop in the above interaction just releasing the Control key?

Can we have a drag and drop without large gridcells? When the balloon is released it won't necessarily end up resting neatly in a grid cell. If the grid cells can be the small steps we have now, we may have a solution.

Giving the user the option to choose their own keys is a great idea. For right-handed folks IJKL may work better than WASD. In my last 3 interviews all 3 users had trouble understanding what WASD was, obviously none of them are gamers. Again @jobara, great suggestions.

[terracoda]

@jessegreenberg Also, nice bunch of testing above. Great discussion and results over all!

[terracoda]

@jessegreenberg @jobara In today's interview with Windows 10 and IE11 (Surface Pro 3 tablet with keyboard) the WASD keys worked sometimes and sometimes they didn't. In a few interactions, only every other key press moved the balloon. And in some cases, there was no movement and then the next press sent the balloon from the the bottom of the sweater all the way to the top in one go.

Some general things:
User kept getting stuck on the main "reset" button - no wrapping to top. Will look more closely at video. Not sure what is causing that.

The Keyboard commands and help dialog only works when balloon has focus. Ideally help should be available from anywhere. I think this was a temporary implementation of help.

Dialog issue: the user ended up back in the content somehow without closing the dialog.

User didn't like that the word bullet was readout by JAWS in the keyboard dialog lists. I'll have to see if there is a way to suppress that naturally. I wonder if that is happening because the content in the dialog has document role or if it is a JAWS detail setting?

This user used a lot of his JAWS hotkeys.

[jobara]

@terracoda in regards to the drag and drop of the grid reorder demo. Yes the "ctrl" is what grabs and releases the item. This demo is specifically about moving an item around a grid. I believe you could have the drop targets be anything really. We have a few other demos as well list, image, and layout. The layout demo might be the most interesting to look at, as the drop targets depend on how many panels are in each column. Also note that in all of these examples, the drop target isn't necessarily where the draggable item is dropped. In these examples we incase the "target" size by actually picking the closest target to where the item is actually released. Hopefully this will be helpful for thinking about interactions for balloons and static electricity.

@terracoda in regards to the issues you saw on the Surface Pro 3 when user testing, I wonder if they are performance related. I have a Surface here running Windows 10, maybe we could try to replicate what you saw, when you are in Toronto.

[terracoda]

@jobara thanks for more links to examples. The interaction I originally proposed is conceptually a drag and drop: one key to grab (and drag) and one key to drop (release). The only issue was technically implementing the simple interactions. And we got side tracked by application role and screen reader modes. This conversation has brought us back around to the essentials and this is great.

Re the delayed dragging on the Surface Pro 3, I agree that it is likely performance related. I haven't seen such behaviour on any other computer. Yes, let's test things when I am in Toronto. I will be at the IDRC Feb 1st to Feb 5th.

[jessegreenberg]

More excellent examples @jobara. These are very helpful, thank you!

I still think the application role will be very important for us. For example, I can't get these examples to work unless the screen reader is forced into 'forms' mode, which is exactly the issue we are facing in Balloons and Static Electricity. It seems like a new question is: How important is it that the screen reader toggle to 'forms' mode automatically? Would it be acceptable to explicitly tell the user to switch modes?

[terracoda]

@jessegreenberg I agree that this is a good question. We are investigating what things can be done implicitly and what has to be explicit. Usually switching to forms mode is an implicit thing that happens and the AT tells the user it has happened, usually with a pop (JAWS makes a popping sound).

[terracoda]

By the way, the whatsock website by Bryan Garaventa has had a re-design. It is much easier to read now.

[terracoda]

@jessegreenberg I think what we need is an

Unassociated Drag and Drop.

See section 2.5 of whatsock website.

[jessegreenberg]

I agree @terracoda, the optimal solution would involve allowing the transition to occur implicitly.

[terracoda]

@jessegreenberg, quoting Whatsock:

The first thing to note, is that the attributes aria-grabbed and aria-dropeffect have no functional effect on the draggability of a component. This must be programmed in advance using JavaScript and CSS to handle mouse and keyboard interaction. The drag and drop ARIA attributes simply convey these relevant behaviors to Assistive Technologies.

My question, is it possible we are not "conveying" the the relevant behaviours correctly?

My initial observations on comparison the with Whatsock code. In terms of roles, there is no talk of the application role for a draggable object, but a draggable object "must have a valid role".
Based on a comparison of the currently implemented code and the code from Whatsock, we may have 3 issues:

1. A valid role on the actual draggable object. Whatsock suggests role="button".
2. A missing aria-grabbed="false" attribute. The draggable="true" attribute is an HTML5 attribute, not an ARIA attribute. I'm not sure, but I've read elsewhere that screen readers began to support aria attributes before HTML5 semantics. According to Whatsock, the aria-grabbed attribute "should be added to either the draggable container element, or to the drag handle element, whichever is designed to be focusable."
3. The draggable div wrapper is not wrapping the actual balloon content, which doesn't look right to me.

Taking your code and the Whatsock code, we might try the following:

<div class="Balloon" id="balloon-14-39-42-275-349-350-353" tabindex="0" role="button" draggable="true" aria-grabbed="false" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349">
<h3 id="balloon-label-349">Yellow Balloon</h3>
<p id="balloon-description-349">Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.</p> 
</div> 

@jessegreenberg, given what you've tried thus far, does this code fill any gaps?

[jessegreenberg]

@terracoda, thank you for going through this resource. This code does fill some gaps. We have not tried placing the draggable and aria-grabbed attributes on the container element. I will give this a try and see how it behaves without the application role for the same platforms tested above.

[jessegreenberg]

He used the JAWS hotkey for Regions to navigate to the Balloon application Region. He immediately heard that he was in forms mode (from JAWS), then hit Enter to enter the form and then he continued.

Very interesting! That is good to know, thanks @terracoda.

In the example above, the books are dragged by the application, not by the user, so the arrows are not needed in this example.

Yes, that is true. These examples do not require the arrow keys for functionality. We have yet to encounter an example where the arrow keys are used to drag an element from one location to another outside of a grid.

I wouldn't expect any dragging announcement to be made, but I would expect an announcement from AT about when the state changes from ungrabbed to grabbed, thus ready for dragging.

That is what I expected too! But that does not seem to happen. When I navigate to a book in Bryan Garaventa's example in with NVDA and JAWS, I hear something like this:

"Move The Last Unicorn to cart draggable droppable"

If you navigate away from the book and then back to it while it is still being dragged, the screen reader announces something like:

"Move The Last Unicorn to cart droppable"

I think that is why the aria-live region for 'dragging' is necessary. It seems like the aria-grabbed and aria-dropeffect provide information upon navigation. No update is announced about the state of the object.

[jessegreenberg]

what is important about the items being contained in a list is that by being in a list, they have the implicit role "listitem"! According to the ARIA documentation an element must have a valid role in order to be able to use aria-grabbed and aria-drop-effect.

Yes that is a good point. Using a valid role in combination with aria drag and drop attributes will be important.

[jessegreenberg]

I recently tested with the balloon represented by the following HTML:

<div tabindex="0" role="button" id="balloon-14-35-38-271-344-347" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-14-35-38-271-344-347" aria-describedby="balloon-description-14-35-38-271-344-347" aria-dragged="false">
    <h3 id="balloon-label-14-35-38-271-344-347">‪Yellow Balloon‬</h3>
    <p id="balloon-description-14-35-38-271-344-347">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

I tested with the same platforms with both WASD and Arrow keys. The aria-grabbed makes it so that 'draggable' is read in almost every screen reader. But keyboard dragging only worked for one platform. Here is the summary of test results:

Testing WASD keys:
NVDA + Chrome:
'Draggable' is read in description, WASD keys do not work

NVDA + Firefox:
'Draggable' is read as part of description, WASD keys do not work

NVDA + IE:
'Draggable' read at very end of description, WASD keys do not work

NVDA + Edge:
'Draggable' NOT read at end of description, WASD keys donot work

JAWS + Chrome:
'Draggable' read twice, WASD keys do not work

JAWS + Firefox:
'Draggable' read twice, WASD keys do not work

JAWS + IE:
'Draggable' read twice, WASD keys do not work

JAWS + Edge:
'Draggable' NOT read, WASD keys Work. Many other descriptions were not functional

Testing Arrow Keys:
JAWS + Chrome:
'Draggable' Read, arrow keys do not work

JAWS + Firefox:
'Draggable' Read, arrow keys do not work

JAWS + IE:
'Draggable' Read, arrow keys do not work

JAWS + Edge:
'Draggable' NOT read, arrow key event passed to browser, but 'keyup' event fired immediately by accessibility API
Drag and drop does not work

NVDA + Chrome:
'Draggable' read, arrow keys do not work

NVDA + Firefox:
'Draggable' read, arrow keys do not work

NVDA + IE:
'Draggable' read, arrow keys do not work

NVDA + Edge:
'draggable' NOT read, arrow keys do not work

[jessegreenberg]

By switching the role to application, we have the following html representation for the balloon:

<div tabindex="0" role="application" id="balloon-14-35-38-271-344-347" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-14-35-38-271-344-347" aria-describedby="balloon-description-14-35-38-271-344-347" aria-dragged="false">
    <h3 id="balloon-label-14-35-38-271-344-347">‪Yellow Balloon‬</h3>
    <p id="balloon-description-14-35-38-271-344-347">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

With this representation, I observe the same results as #116 (comment) and #116 (comment)

[terracoda]

@jessegreenberg Are you sure aria-dragged is a valid aria attribute? I can only find it on the MDN page, not in the spec.

[terracoda]

Also, @jessegreenberg that's a lot of testing and sadly not a lot of working keys :(

[jessegreenberg]

Are you sure aria-dragged is a valid aria attribute? I can only find it on the MDN page, not in the spec.

Great catch @terracoda! Totally my mistake. I initialized aria-grabbed correctly, but accidentally set aria-dragged when linking to the model.

I don't think that this will have changed the screen reader behavior too much, but I will fix and retest on the platforms listed above.

[jessegreenberg]

Also, @jessegreenberg that's a lot of testing and sadly not a lot of working keys :(

Yes, unfortunately :(. Our best results so far involve the application role and the WASD keys. Application role and arrow keys work, but only with JAWS + IE.

[jessegreenberg]

I just tested this. The interaction with the keyboard behaves exactly as it does above. However, there is one difference:

For NVDA + Firefox with role=application, NVDA announces when the balloon toggles from aria-grabbed=true to aria-grabbed=false in exactly the way that @terracoda suggested in #116 (comment)

This does not happen for any other browser or with JAWS.

[terracoda]

Well, that's good that at least one browser/AT combo announces the state. As we learned today from Joseph, aria-grabbed and aria-dropeffect have not been implemented well.

[jessegreenberg]

It seems like either the browsers or the AT are not handling the application role correctly. There are two steps to follow through to see if we can get the desired keyboard navigation:

• Investigate the Rendered accessibility tree using tools listed in Determine best accessible representation of Balloon #108 (comment) to see if there are any red flags that would prevent the desired behavior
• Reproduce the issue in a simplified context, prepare bug reports referencing this context to the browsers.

[terracoda]

@jessegreenberg I agree that we need to investigate the the Accessibility Tree and simplify the context.
Would you mind posting the rendered HTML as it is in Instance 11? I know it is slightly different from my prototyped PDOM.

Here are the few approaches we might take based on discussions with Joseph Schuehammer and some post interview thoughts.

1. Reduce nesting of code structure. Doing this should reduce the duplication of content. This might involve not using the H3 and aria-labelledby structure for the Balloon (not 100% sure, though).
2. Alternatively, we can try using a span element instead of a div as a container. A span is an inline element (not a block element) and may behave differently with the role application than a div.
3. We can experiment with input type=number (which can be associated with text strings). The Balloon does indeed have a value (a net zero or net negative charge), so input type=number works well semantically. This may also have the added benefit of addressing a description issue that some participants raised in the interviews. With this input type, I think we briefly looked at the idea of using aria-role=spinbox.
4. One other idea is to change the interaction flow by adding an explicit grab before the drag. I think this was my original idea way back when. A comment from one of the participants supports this interaction pattern. He mentioned that in gaming, he often uses the Spacebar to pickup AND drop things.
   Would a change in the design of the interaction successfully push the screen reader into forms mode and then the application can deal with the drag interaction after that.
5. Finally, need to look into custom labeling possibilities with the the ARIA specification. Can we tell a screen reader to read out the balloon as simply as, "Interactive Balloon" using an ARIA technique?

Example with explicit grab interaction (number 4):

• user moves focus to balloon with the Tab key and hears "Play with Balloon, button".
• user presses Spacebar (or Enter) to play with the Balloon.
• Nav/interaction cue is read out. "Use W, A, S...to drag balloon up, left.... Press Spacebar to release."

So, I am wondering if an explicit grab might improve the interaction for keyboard (and screen reader users) even though it is an extra key press? The reason behind it is that the user would get the instructions for the interaction in two steps which might be more understandable.

[jessegreenberg]

Sure @terracoda, here is the up to date PDOM for BASE:

<div class="accessibility">
    <div class="ScreenView" id="screen-view-17-42">
        <header role="banner" aria-labelledby="scene-label-17-42">
            <h1 id="scene-label-17-42">‪Balloons and Static Electricity‬</h1>
        </header>
        <main class="ScreenViewContainer">
            <section>
                <h2>‪Scene Summary‬</h2>
                    <p>‪This simulation contains a Play Area and a Control Panel. The Play Area is a room. It has a balloon, a sweater and a wall. The Control Panel allows you to add a balloon, remove wall and reset entire experiment. Currently, sweater and wall have many pairs of negative and positive charges, the balloon, just a few. Balloon is in middle of Play Area, evenly between sweater and wall. Tab to navigate to next object. Press question mark for keyboard commands and help.‬</p>
            </section>
            <section id="play-area-17-42-45" aria-labelledby="heading-node-44">
                <div id="sweater-17-42-45-46" aria-live="polite" aria-labelledby="sweater-label-17-42-45-46" aria-describedby="sweater-description-17-42-45-46">
                    <h3 id="sweater-label-17-42-45-46">‪Sweater‬</h3>
                    <p id="sweater-description-17-42-45-46">‪The sweater has a ‪neutral‬ charge with ‪no‬ more positive charges than negative ones.‬</p>
                </div>
                <div tabindex="0" role="application" id="balloon-17-42-45-278-351-354" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351-354" aria-describedby="balloon-description-17-42-45-278-351-354 navigation-description-17-42-45-278-351-354">
                    <h3 id="balloon-label-17-42-45-278-351-354">‪Yellow Balloon‬</h3>
                    <p aria-live="assertive" id="balloon-description-17-42-45-278-351-354">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
                    <p id="navigation-description-17-42-45-278-351-354">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
                </div>
                <div tabindex="0" role="application" id="balloon-17-42-45-278-279-282" draggable="true" aria-grabbed="false" class="Balloon" hidden="" aria-labelledby="balloon-label-17-42-45-278-279-282" aria-describedby="balloon-description-17-42-45-278-279-282 navigation-description-17-42-45-278-279-282">
                    <h3 id="balloon-label-17-42-45-278-279-282">‪Green Balloon‬</h3>
                    <p aria-live="assertive" id="balloon-description-17-42-45-278-279-282">‪Green Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
                    <p id="navigation-description-17-42-45-278-279-282">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
                </div>
                <div id="wall-17-42-45-164" aria-labelledby="wall-label-17-42-45-164">
                    <h3 id="wall-label-17-42-45-164">‪Wall‬</h3>
                    <p id="wall-description-17-42-45-164">‪The wall has a neutral charge, no more positive charges than negative ones.‬</p>
                </div>
            </section>
            <section id="control-panel-17-42-423" aria-labelledby="heading-node-424">
                <h2 id="heading-node-424">‪Control Panel‬</h2>
                <div id="element-container-440">
                    <button aria-describedby="441">‪Remove Wall‬</button>
                    <p id="441">‪Toggle to conduct experiments with or without the wall.‬</p>
                    <p id="442" aria-live="assertive" aria-hidden="true">‪Wall added to Play Area‬</p>
                </div>
                <div id="toggle-container17-42-423-473-467-454-449" aria-live="polite">
                    <button id="abswitch-17-42-423-473-467-454-449-453" aria-pressed="false">‪Two Balloon Experiment‬</button>
                    <p aria-hidden="true">‪Green balloon removed from Play Area‬</p>
                </div>
                <div id="element-container-465">
                    <button>‪Reset Balloon‬</button>
                    <p id="466">‪Select to ‪Reset Balloon‬ to initial position and an uncharged state.‬</p>
                </div>
                <input type="reset" value="‪Reset All‬" tabindex="0">
            </section>
        </main>
    </div>
</div>

[terracoda]

@jessegreenberg Thanks!

[jessegreenberg]

Reduce nesting of code structure.

From above, here is the representation of the balloon:

<div tabindex="0" role="application" id="balloon-17-42-45-278-351-354" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351-354" aria-describedby="balloon-description-17-42-45-278-351-354 navigation-description-17-42-45-278-351-354">
    <h3 id="balloon-label-17-42-45-278-351-354">‪Yellow Balloon‬</h3>
    <p aria-live="assertive" id="balloon-description-17-42-45-278-351-354">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
    <p id="navigation-description-17-42-45-278-351-354">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

I am not sure how to reduce nesting here. Perhaps we can try it without descriptions and labels and see if the balloon behaves any better. We can then try to add descriptions and attributes back in to the balloon to see what is causing the repetition.

Alternatively, we can try using a span element instead of a div as a container.

I did not quite understand this, can a span exist on its own without being wrapped by another element? It seems strange that a widget would be represented by a span element. Also, I don't think inline elements can contain block elements like p.

We can experiment with input type=number (which can be associated with text strings).

I will give this a shot! We will see how browsers behave with this and how AT's read this element. My initial concern is that an input of type number that represents charge could be confusing when the user is actually controlling the position.

One other idea is to change the interaction flow by adding an explicit grab before the drag.
Would a change in the design of the interaction successfully push the screen reader into forms mode and then the application can deal with the drag interaction after that.
Example with explicit grab interaction (number 4):

user moves focus to balloon with the Tab key and hears "Play with Balloon, button".
user presses Spacebar (or Enter) to play with the Balloon.
Nav/interaction cue is read out. "Use W, A, S...to drag balloon up, left.... Press Spacebar to release."

Sure, we could add an explicit 'grab' interaction. A two step interaction could be helpful. When would the user hear the descriptions about the balloon charge/location?

[terracoda]

@jessegreenberg
Regarding the nesting: that was likely caused by the form control that was rendered in earlier versions. I agree, I don't think we can reduce nesting - there is no nesting now :-)

Regarding duplication, as discussed today, this might be caused by the screen reader reading out the content once as aria-labelledby / aria-describedby and once as actual content. Not sure how to handle this problem. Maybe dynamically manage the attributes (labelledby and describedby), but that is just a guess.

Regarding a new two-step interaction, and when to read charge and position information, I have to think this through a little, but I think we'll need a super brief description upon grab, and then upon release something a little more verbose. The "Grab Balloon" button label would hopefully be read out last, after the release description.

[jessegreenberg]

#116 (comment)

After a recent update to firefox, Arrow keys are now supported by NVDA + Firefox on my Windows 10 Machine!! No change to the current balloon implementation.

In general, accessibility is performing much better on this platform after this update. I would like to know why.

[jessegreenberg]

Just tested with JAWS, this AT exhibits the same issues on all platforms. Event goes to the browser, but 'keyup' fired immediately after 'kedown' preventing dragging behavior.

[jessegreenberg]

Alternatively, we can try using a span element instead of a div as a container.

Replacing div with span does not seem to have an impact on screen reader behavior either way.

[jessegreenberg]

One other idea is to change the interaction flow by adding an explicit grab before the drag.

Creating a separate issue for this.

[jessegreenberg]

From #116 (comment)

We need to investigate the aria application role. Our understanding is that this role should allow all key events to be passed to the web application. This has not been our experience.

We have learned a lot about the application role since ths issue was first opened. Primarily, we now understand that it is up to the AT to determine which events are passed to the browser and which ones aren't. This makes it more challenging to use certain keys for interactive behavior, but we can generally expect certain keys (such as WASD) to be handed to the browser with the application role. Our next task will be to submit a bug report to freedom scientific to understand why keydown events from the arrow keys are not supported with role=application. I will open a special issue in this repo to track this.

This issue has gotten a bit lengthy, and I think we have learned what we needed from the discussions and tests above. Closing, and further investigation with application role will be reduced to more specific issues.