Experiment with updating the accessibility peer bounds

[phetsims]

Right now peers are just at top:0, left: 0, but some accessibility technologies may require the correct locations for the peers. We should investigate adding this support, with the caveat that it may harm performance, especially for accessible play area objects that move every frame, and especially on tablets.

[ariel-phet]

assigning to @jessegreenberg for issue hygiene, can this be closed? (especially since you seem to have a clear a11y strategy these days)

[jessegreenberg]

Many AT, including touch devices, need the correct locations of the peers. This will still need to be addressed since the parallel DOM is still at top:0 left:0. We have seen some users try to find elements on the screen by waving the cursor around like a wand. Since the locations in the parallel DOM were incorrect, the elements could not be found. I expect this could have some performance implications.

[jonathanolson]

Correct about the performance implications.

We should probably collaborate when it comes time to address this. Also let me know if it's possible to "detect" when we need the positions to be accurate (i.e. if we can leave them un-positioned until the user interacts with accessibility tools, that would be good for performance).

This is mostly only a performance implication for when a displayed element moves (not concerned about startup/screen switches/etc.)

[jessegreenberg]

We are ready to investigate adding this for accessibility. @jonathanolson would you be able to help work on this? I will need some help integrating this into Scenery.

This could improve discoverability of sim components and add add navigation strategies for touch and mouse devices with a screen reader enabled. There are a few things that are not immediately obvious to me about DOM styling:

• Can every element be correctly styled to support this? (<input type=range> and other composite elements seem problematic)
• Can DOM elements be discoverable without receiving all events? We want this styling to handle discoverability but we want the scenery event system to handle all other events.
• Is there other DOM styling (outside of the parallel DOM) that is already in place that needs to be modified for this?

[ariel-phet]

@jessegreenberg lets mark this as low priority for now, please bump up to medium priority once @jonathanolson is working on make-a-ten as his main project (tentatively within the next couple of weeks)

[jonathanolson]

Here's a rough outline (by @jonathanolson and @jessegreenberg) for the general implemetation for positioning peers:

Assumptions:

• Each peer has a width and height, and its normal bounds go from the upper-left corner (0,0) to (width,height)
• We can find a way to listen to how the peer's untransformed bounds change, so that we can just the transform (see http://stackoverflow.com/questions/6492683/how-to-detect-divs-dimension-changed)
• We want to have containers (non-leaf accessible instances) have highlights
• We need the containers themselves sometimes to be highlighted (can't highlight random DIVs)

Implement:

AccessibleInstance will get nodesToParent, which will be an Array of nodes from (not including) the parent accessible instance's node to (including) our accesible instance's own node. e.g. if our parent's trail is [ root, a, b, c ], and our trail is [ root, a, b, c, d, e, f ], our nodesToParent will be [ d, e, f ].

We'll compute nodesToParent on initialization (as it won't change, as those trails are immutable).

We'll compute a peerToLocalMatrix that will transform our peer's rectangle (0,0,peerWidth,peerHeight) into our localBounds rectangle (minX, minY, maxX, maxY). This is:

// inefficient version, can potentially combine later to not create extra matrices that aren't used
var localBounds = node.localBounds;
var clientWidth = element.clientWidth;
var clientHeight = element.clientHeight;
Matrix3.translation( localBounds.minX, localBounds.minY ).multiplyMatrix( Matrix3.scale( localBounds.width / clientWidth, localBounds.height / clientHeight ) );

We'll make an inverted version of it available to our child accessible instances.

Thus the transformation matrix needed for each peer (assuming nesting) will be:
( inverted parent's peerToLocalMatrix ) * nodesToParent[ 0 ].getMatrix() * ... * nodesToParent[ n - 1 ].getMatrix() * ( our peerToLocalmatrix )

Ideally we can use as much mutability as possible to prevent allocations (no allocations would be perfect). Can create a few matrices per accessible instance that can be mutated for those purposes.

Then use getCSSTransform to set peerElement.style.transform. (peerElement.style[ Features.transformOrigin ]).

Peers should have the following to set CSS (which is needed for proper display):

element.style.position = 'absolute';
element.style.top = '0';
element.style.left = '0';
element.style.transformOrigin = 'left top';

or equivalent with browser prefixes. Potentially use Features.transform and Features.transformOrigin, e.g. element.style[ Features.transform ].

Invalidation:

Each accessible instance should have a "dirty" and "childrenDirty" flag related to this (possibly more specific names).

When an accessible instance is "invalidated", it sets this.dirtyFlag = true (naming), and childrenDirty (or equivalent) on every ancestor (up the tree).
e.g.:

invalidate_peer_transform() {
  if ( !dirty ) {
    dirty = true;

    var node = parent;
    while ( node ) {
      node.hasDirtyChild = true;
      node = node.parent;
    }
  }
}

Every frame, to update, scan in a depth-first manner through every instance that has childrenDirty=true, looking for instances with dirty=true.
Update instances with dirty=true first (update their transforms) before proceeding to their children, as parents need to be updated before their children (due to peerToLocalMatrix).

Transform listeners should be added to every member of nodesToParent, e.g. node.onStatic( 'transform', .... ). This should invalidate JUST the specified accessible instance

localBounds listeners should be added to the node of the accessible instance. When this is triggered, it should invalidate this accessible instance AND all of its direct children (but NOT the entire subtree). This is because our children use our peerToLocalMatrix).

NOTE: localBounds listeners will AT LEAST fire at or before the validateWatchedBounds call in Display.updateDisplay. Ideally we should do the depth-first scan and transform style update near the BOTTOM of Display.updateDisplay.

We should listen to our peer DOM element's bounds, and when invalidating, do the same as if the localBounds listener was triggered (mentioned above).

[jessegreenberg]

We can find a way to listen to how the peer's untransformed bounds change, so that we can just the transform (see http://stackoverflow.com/questions/6492683/how-to-detect-divs-dimension-changed)

I will take a look at this part in particular before we implement. It would be nice if we could cleanly listen for this.

[jessegreenberg]

The stackoverflow link recommends this: http://marcj.github.io/css-element-queries/

This is a library that could help detect DOM element bounds changes. The library uses ResizeSensor.js, which detects resizing DOM elements. css-element-queries allows you to add specific css properties to detect styling changes, and I don't think we want to use this. But it might be possible to use only .

In some testing, ResizeSensor.js does work, but only when you modify the styling. If at any point, you modify the textContent, ResizeSensor will fail to detect changes to the element's bounds. Next step will be to look at ResizeSensor to figure out why.

For example, here is a sensor:

  new ResizeSensor( testButton, function() {
    console.log( 'Changed to ' + testButton.clientWidth );
  } );

A fire function is added to a button which changes its textContent.
This works just fine:

  var fire = function() {
    testButton.style.width = testButton.clientWidth + 10 + 'px';
  }

And both of the following prevent ResizeSensor from firing its callback.

  var fire = function() {
    var randomString = sentences[ Math.floor( Math.random() * sentences.length ) ];
    testSubject.textContent = randomString;
  }

  var fire = function() {
    var randomString = sentences[ Math.floor( Math.random() * sentences.length ) ];
    testSubject.textContent = randomString;
    testButton.style.width = testButton.clientWidth + 10 + 'px';
  }

[jessegreenberg]

Perhaps instead of using ResizeSensor, we could use a MutationObserver to listen for things that might change bounds rather than listening to a bounds change directly?

Support for MutationObservers is pretty good at this point (http://caniuse.com/#feat=mutationobserver)

[jessegreenberg]

ResizeSensor uses RequestAnimationFrame to observe changes. MutationObserver is purely event based and might perform better.

[jessegreenberg]

The MutationObserver allows you to specify a configuration so that you can listen to changes to text content, child nodes, and attributes. You can even specify names of specific attributes for which mutations need to be observed.

[jessegreenberg]

I think MutationObserver should work, do you agree @jonathanolson?

[jonathanolson]

MutationObserver sounds good to me.

[jessegreenberg]

I took an initial stab at this on branch accessible-peer-bounds (see #705). I went through the implementation steps listed in #41 (comment), also using MutationObserver to listen for when a DOM element's untransformed boudns change, and I am amazed with how well it is working! I can navigate the accessible DOM and activate sim elements, including complex things like elements that use drag handlers. Many thanks to @jonathanolson for the detailed list! I have not begun the "Invalidation" portion nor considered performance too much yet, initial commits are to investigate whether or not this will work at all. Here is what I have found so far:

• In order for events to reach the sim when VoiceOver was enabled, I had to remove domEvent.preventDefault() for touchStart and touchEnd in Input.js.
• Now that DOM elements have non-zero bounds, the browser is sometimes shifting Display content around the window to make the focused HTML element viewable on screen. We need to find a way to prevent the browser from doing this while still keeping accessible content available for VoiceOver and other AT.
• After changes, performance seems great on Chrome and iPad3 Safari (Amazing!). But sims fail to load in Edge, Firefox, and IE11. I have not tested slower ipads yet either. We will need to test performance on a number of platforms, investigate performance improvements. If not possible, we will want to just enable these changes for Safari, since it seems to support this. If that is not possible we will need to completely reconsider this approach.
• DOM bounds aren't perfectly around Node bounds yet, they will need to be more accurate.

[zepumph]

@jessegreenberg @emily-phet @mbarlow12 and @zepumph spoke about this issue in regards to a11y infrastructure work scheduled for this summer. There seems to be a lot of considerations with this issue that are hard to tag a direct priority.

@emily-phet hopes that we can make some progress on this. We will need to balance this with other issues.

At the very least we should do a bit of investigation to determine how reasonable work on this issue is for the coming season.

[jessegreenberg]

Removing my assignment until we continue work on this (possibly later this summer), though we should still discuss at the next a11y-dev meeting.

[jessegreenberg]

I have been exploring this as a solution to mobile accessibility. #835 is the branch with progress thus far. So far it is working great, but performance is pretty rough. In sims I have tested on my pretty fast machine using Chrome, what was previously animating at 60 fps, is now animating at 30fps. Next step will be to try the "Invalidation" approach listed in #41 (comment).

[jessegreenberg]

I made #852 to continue this work, and we will track more specific questions there.