[Spec] Accessibility Talkback and VoiceOver Properties

[PureWeen]

Semantic Properties for Accessibility

This defines the first set of semantic properties we want to expose for mapping xplat properties to native accessibility properties. In general our approach for accessibility within MAUI core is to stay away from "faking it" or trying to force equal behavior between the platforms. By default the native platforms do a very good job of naturally being accessible. The best thing .NET MAUI can do is to stay out of the way as much as possible but then easily allow users to tweak what they need in order to meet WCAG certification.

API

interface IFrameworkElement

public interface IFrameworkElement
{
     SemanticProperties SemanticProperties { get; }
}

Class SemanticProperties

https://github.com/dotnet/maui/blob/main/src/Core/src/Primitives/Semantics.cs
https://github.com/dotnet/maui/blob/main/src/Controls/src/Core/SemanticProperties.cs

Properties

API	Description
Description	iOS: accessibilityLabel, Android: contentDescription
Hint	iOS: accessibilityHint, Android: hint
HeadingLevel	iOS: AccessibilityTraits > header, Android: accessibilityHeading, WinUI: AutomationProperties.HeadingLevelProperty
LabeledBy	See #845
IsInSemanticTree	See #927

Implementation Details

See #635 for initial set of changes to be introduced in P3.

On Description and Hint

accessibilityHint on iOS and hint on android aren't perfectly symmetrical in behavior. The property on iOS is very specific for accessibility whereas the hint property on Android behaves slightly differently depending on the control you apply it to. Android controls without text values like EditText/Switch/Checkbox will display the hint with the control, and also manipulate other accessibility properties (contentDescription) (whereas with controls with text values, the hint is not displayed and is read after the text value / contentDescription / etc). This feels acceptable because it allows the user full control over what the native property will get set to.

With Name/HelpText we currently have various permutations that will concatenate the values or do various other permutations based on the type of control. This type of "we know best" approach has mainly led to a very confusing experience for users. Natively the way TalkBack devices use hint is very specific. Hints are meant to provide extra context for what the control is/does, but not necessarily how the user should interact with it. Concatenating HelpText and Name confuses the purpose and doesn't really map directly to native APIs.

I realize that the 'Hint' property will conflict with the 'Entry.Placeholder' property implementation on Android, since that also maps to the 'Hint' property, but setting a different 'Hint' than placeholder isn't really a recommended approach. You should always make the experience of your app as symmetrical as possible for users. If someone wants to make the 'Hint' text different from the 'Placeholder' text, then it's on them to break from recommendation. At a later point we will add additional APIs so users can detect if VoiceOver/TalkBack is enabled to allow them to make more informed decisions

On Heading

Whereas WinUI offers 9 levels of headings, Android and iOS offer the ability to simply set a control as a heading. Therefore, we will be offering HeadingLevel - when any HeadingLevel is specified on .NET MAUI, it will map to the appropriate WinUI HeadingLevel specified, and default to setting the heading property to true on Android and iOS.

Scenarios

XAML Example

The SemanticProperties class will be defined inside of Maui.Core but each UI framework can choose to expose these properties out however it makes sense to

<StackLayout x:Name="thing" SemanticProperties.Hint="Click to access" />

Todo Investigation Points

• Android only provides the Hint property on controls that inherit from TextView so what should be the expectation if users set Semantics.Hint on ProgressBar, layouts, ActivityIndicator, etc..
• Pre API-26 on Android does not read out the hint if you have contentDescription set. You can make a TextView (ILabel) control work the same but controls like Button can't be made the same between Pre and Post API 26. Post API 26 will read the text as "" "" "". This is a recognized bug by Android and they have a few cases in the source code TextInputLayout where they workaround this limitation.
• Do we want to take these features a bit further and let people articulate more about the decisions made with the Hint and Description property. For example, we could add an Enum indicating that a Hint should be concatenated where there is no platform equivalent.
• We could also go really far and build out a more behavior oriented semantics class/semantics service that let's people describe intent more thoroughly

Difficulty : [medium]

[pictos]

Would be good if we can add Commands for a11y. For the case that I want to do perform an action just for a11y scenarios.
For example, imagine that I have a slider button and for a11y I want to use the LongPress gesture to perform the same action as the slider action.

[PureWeen]

@pictos do you by chance have some native documentation or samples you can provide? I think that would help us understand the use cases better

[AmrAlSayed0]

The XAML shows accessibility properties as attached properties, how would they be represented in the new architecture?

[PureWeen]

@AmrAlSayed0 I updated the spec a bit and included an interface

It'll basically just be a property on IView and then XAML will just construct an AccessibilityProperties class from the attached properties that gets passed back to Core

How APIs are exposed from XAML don't really have much influence on how they are exposed from Maui.Core

[pictos]

@PureWeen that is nothing too fancy you already have the implementation [here] for one-click (https://github.com/PureWeen/A11YTools/blob/main/A11YTools/A11YTools.Android/AndroidA11yService.cs#L57).
The idea is to create a long press handler. So the use case will be

<StackLayout x:Name="thing" Accessibility.Hint="Click to access"
Accessibility.Command="{Binding OneClickCommand}" 
Accessibility.LongPressCommand="{Binding LongPressActionCommand}"/>

This is better than adding a GestureRecognizer because in some scenarios the a11y behavior is very specific.

[GalaxiaGuy]

@pictos Are you intending that command to only be available if VoiceOver/TalkBack is running?

I've implemented something like that recently on Android, but on iOS I used accessibilityCutomActions instead.
https://developer.apple.com/documentation/uikit/uiaccessibilitycustomaction

[pictos]

@GalaxiaGuy thanks for the docs, my team are using the approach that I described in our project, we used the Shane code base, which is working like a charm and fixes tons of bugs related to a11y

[LeonarddeR]

Is it me or is this issue particularly aimed at Android and iOS? As for Windows, I think it would be interesting to know how to tweak UIAutomation element behavior.

[brunoprietog]

@PureWeen Is it possible to implement an api to assign custom actions to the controls? As referenced by @GalaxiaGuy. I imagine something like React Native's custom actions. Voiceover and Talkback have them.