[Spec] Implement IsInSemanticTree property for accessibility

[rachelkang]

Implement IsInSemanticTree property for accessibility

Create a property that determines whether a specific element (and its children) are in the semantic tree for accessibility. This influences whether or not screen readers can access and read aloud in-app elements.

This property is reminiscent of the Xamarin.Forms AutomationProperties.IsInAccessibleTree property. However, IsInSemanticTree goes beyond the boolean and with a greater scope, will enable developers to have more control over the accessibility of each element in their apps.

API

SemanticProperties.IsInSemanticTree

Properties

API	Description
IsInSemanticTree	iOS: isAccessibilityElement, accessibilityElementsHidden; Android: importantForAccessibility

public enum IsInSemanticTree
{
     Default,
     IncludeNode,
     RemoveNode,
     RemoveSubtree
}

IsInSemanticTree.Default
Every .NET MAUI control will have a Default value of being in the semantic tree (IncludeNode) or being outside of the semantic tree (RemoveSubtree). Developers will be able to use this property in complex/nuanced scenarios where straying from the defaults will enable them to make their apps more accessible (see example below).

IsInSemanticTree.IncludeNode
IncludeNode ensures the specified element is in the semantic tree and will be accessible to screen readers. It is the opposite of RemoveNode.

IsInSemanticTree.RemoveNode
RemoveNode ensures the specified element is NOT in the semantic tree and will NOT be accessible to screen readers. It is the opposite of IncludeNode.

IsInSemanticTree.RemoveSubtree
RemoveSubtree ensures the specified element AND any children it may have are NOT in the semantic tree and will NOT be accessible to screen readers. This is the equivalent of setting RemoveNode to the specified element, and every single one of the specified element's child elements.

Implementation Details

Enum	iOS	Android
IncludeNode	isAccessibilityElement=true	importantForAccessibility=yes
RemoveNode	isAccessibilityElement=false	importantForAccessibility=no
RemoveSubtree	accessibilityElementsHidden=true	importantForAccessibility=noHideDescendants

Scenarios

XAML Example

<Label Style="{DynamicResource Glyph}" Text="&#xf388;" SemanticProperties.IsInSemanticTree="RemoveNode" /> // renders heart icon
<Label Text="Heart" />

The above sample renders two labels - one as a heart icon, and one as the text "Heart". By default, all labels are in the semantic tree and therefore screen reader accessible. Therefore, without the setting of SemanticProperties.IsInSemanticTree="RemoveNode", screen readers would focus on both the icon and the text. In this case, where the text already describes the icon, it is unnecessary to focus on the icon at all (by default, the screen reader would focus on it and confusingly read back nothing; if a semantic description is set to it, it would read back information that is already captured by the text, thereby making it redundant). In this scenario, manipulating whether or not a control IsInSemanticTree allows for a more accessible experience.

Further considerations and areas of investigation

• Nested controls
  • With layouts: My current thinking here is that IncludeNode/RemoveNode just won't affect anything; RemoveSubtree will just make all the children inaccessible, essentially applying RemoveNode to all of the children individually.
  • When RemoveSubtree is set on a parent and IncludeNode is set on any child, which will take precedence? My current thinking is that in this scenario, the RemoveSubtree on the parent will take precedence and all the children will be set to RemoveNode no matter what.
  • Semantic grouping APIs will be further deliberated independent of this property
• Interaction with other SemanticProperties
  • Should the setting of other SemanticProperties (i.e. Description, Hint, IsHeading, LabeledBy) influence the setting of IsInSemanticTree? Current thinking here is yes - if any of these properties are set, IsInSemanticTree=IncludeNode by default. For example, an image with SemanticProperties.Description set, will also have IsInSemanticTree=IncludeNode set, despite defaults.
• Interaction with related APIs
  • Keyboard navigation APIs. What should be the relationship between screen reader accessibility via touch navigation vs. keyboard navigation? The answer varies natively from platform to platform. This spec is currently only defined for screen reader accessibility via touch navigation.
  • Related APIs include Android's focusable, focusableInTouchMode, screenReaderFocusable, among others. Current thinking is that these APIs will be exposed separately, if anything, and will not be integrated into this IsInSemanticTree API.

Difficulty: [medium]

Related: #469

[DDSoftDennie]

Love this concept! Accessibility by default.

[llevera]

Couple of thoughts on this:

1. I think it is an important distinction that this property is called IsInSemanticTree and not mention the term Accessible as, because the property has a getter, otherwise it would imply somehow that this reflects the final "decision" by the platform whether this control is read out. This is because that decision is an incredibly complex one, especially on Android, where the implementation itself might end up being done by calling a Set method, with no corresponding Get (such as setFocusable).
2. That is emphasised well by the property have the default of Default as it re-affirms that unless set otherwise, the effect of this property is not defined. Ie, if this property is IncludeNode or RemoveNode or RemoveSubtree, then the underlying platform will have this value forced, otherwise it's value is unknown.
3. It is my opinion that RemoveSubtree is veering into a grouping/merging API and is best left out for the moment. It is certainly not required to support AutomationProperties.IsInAccessibleTree, as this is a boolean

[llevera]

Implement IsInSemanticTree property for accessibility

Create a property that determines whether a specific element (and its children) are in the semantic tree for accessibility. This influences whether or not screen readers can access and read aloud in-app elements.

This property is reminiscent of the Xamarin.Forms AutomationProperties.IsInAccessibleTree property. However, IsInSemanticTree goes beyond the boolean and with a greater scope, will enable developers to have more control over the accessibility of each element in their apps.

As mentioned above, I am not 100% convinced such a enum is the right mechanism to enable more control. Group semantics might be more natural to express the parent-child relationships, although I do concede iOS and Android do expose this in similar ways to what is proposed for IsInSemanticTree.

API

SemanticProperties.IsInSemanticTree

Properties

API Description
IsInSemanticTree iOS: isAccessibilityElement, accessibilityElementsHidden; Android: importantForAccessibility

importantForAccessibility will work for this purpose, and still seems to most reliable despite newer properties. https://developer.android.com/guide/topics/ui/accessibility/principles and https://developer.android.com/reference/android/view/View#attr_android:screenReaderFocusable seem to push strongly for screenReaderFocusable to be used when overriding the element to be screen reader focusable, but leaving it as false does not necessarily remove the element from the tree.

Implementation Details

Enum iOS Android
IncludeNode isAccessibilityElement=true importantForAccessibility=yes
RemoveNode isAccessibilityElement=false importantForAccessibility=no
RemoveSubtree accessibilityElementsHidden=true importantForAccessibility=noHideDescendants

accessibilityElementsHidden only applies to the children. You would also have to set isAccessibilityElement=false on the element to remove itself and it's children.

Further considerations and areas of investigation

• Nested controls

  • With layouts: My current thinking here is that IncludeNode/RemoveNode just won't affect anything; RemoveSubtree will just make all the children inaccessible, essentially applying RemoveNode to all of the children individually.
  • When RemoveSubtree is set on a parent and IncludeNode is set on any child, which will take precedence? My current thinking is that in this scenario, the RemoveSubtree on the parent will take precedence and all the children will be set to RemoveNode no matter what.

This is sensible - on iOS is very difficult for a parent to be removed but with any children included. Doing so requires high levels of workarounds.

• Semantic grouping APIs will be further deliberated independent of this property

• Interaction with other SemanticProperties

  • Should the setting of other SemanticProperties (i.e. Description, Hint, IsHeading, LabeledBy) influence the setting of IsInSemanticTree? Current thinking here is yes - if any of these properties are set, IsInSemanticTree=IncludeNode by default. For example, an image with SemanticProperties.Description set, will also have IsInSemanticTree=IncludeNode set, despite defaults.

The problem with linking these properties is that this is that it will go far further than the native platforms. Eg, if Description is set, then on android, this would therefore cascade to setting importantForAccessibility to be Yes, which is more than what Android would do when you set contentDesc. It would also create potential contradictions only determined by order - for eg if the property IsInSemanticTree is false and Description is set, then the final value of IsInSemanticTree would be non-obviously resolved to true if Description is evaluated last. My opinion would not be to tie these together.

For eg, on iOS this code

label.isAccessibilityElement = NO;
label.accessibilityLabel = @"label"; 

Results in label.isAccessibilityElement = NO, and the element subsequently not being read.

• Interaction with related APIs

  • Keyboard navigation APIs. What should be the relationship between screen reader accessibility via touch navigation vs. keyboard navigation? The answer varies natively from platform to platform. This spec is currently only defined for screen reader accessibility via touch navigation.

By sticking to importantForAccessibility, the keyboard navigation on Android will not be influenced, for neither good or for bad.

• Related APIs include Android's focusable, focusableInTouchMode, screenReaderFocusable, among others. Current thinking is that these APIs will be exposed separately, if anything, and will not be integrated into this IsInSemanticTree API.

Doing so will mean keyboard navigation is not effected. This might be best left for native calls.

[llevera]

My overall thought on IsInSemanticTree, is that it I see the necessity, esp for presenting a way to maintain the function of AutomationProperties.IsInAccessibleTree.

But generally I think it should be kept as a fairly utilitarian pass through to IsAccessibilityElement and ImportantForAccessibility without any intelligence added, and esp not try and abstract a tidier way of manipulating the parent-child relationship. That would be better with semantics that describe grouping, which is almost always what manipulating the accessibility tree at this level is aimed at achieving.

[PureWeen]

After conversations with the team and @llevera we feel that these APIs don't currently make sense for the current MAUI timeline.

We've added the following spec
#1214

That will address mapping to accessibilityElementsHidden=true | importantForAccessibility=noHideDescendants