[iOS|Android] Fix screen reader order #9702

samhouts · 2020-02-25T01:05:57Z

Description of Change

iOS MasterDetailPage will no longer read MasterPage contents when the master page is hidden and will no longer read DetailPage contents when the master page is visible.
iOS can now tab stop on a Frame and can read its contents properly.
iOS/Android screen readers should no longer skip over layouts and should read all children in declaration order and/or TabIndex order.
TabIndex should no longer return elements that are Enabled = false or IsAccessibilityElement = false.

Issues Resolved

API Changes

These are breaking API changes, but they drive me crazy. I will revert if someone has an issue with this.

TabIndexExtensions:

-public static SortedDictionary<int, List<ITabStopElement>> GetSortedTabIndexesOnParentPage(this VisualElement element, out int countChildrensWithTabStopWithoutThis)
+public static SortedDictionary<int, List<ITabStopElement>> GetSortedTabIndexesOnParentPage(this VisualElement element)


-public static IDictionary<int, List<ITabStopElement>> GetTabIndexesOnParentPage(this ITabStopElement element, out int countChildrensWithTabStopWithoutThis, bool checkContainsElement = true)
+public static IDictionary<int, List<ITabStopElement>> GetTabIndexesOnParentPage(this ITabStopElement element, out int countChildrenWithTabStopWithoutThis)

Platforms Affected

Core/XAML (all platforms)
iOS
Android
UWP

Behavioral/Visual Changes

None

Before/After Screenshots

Not applicable

Testing Procedure

Use TalkBack/VoiceOver to test the Control Gallery and make sure everything is read in an expected order.
Use keyboard to test that TabIndex is still functioning as expected.

PR Checklist

Targets the correct branch
Tests are passing (or failures are unrelated)

fixes #4220

arevellfraedom · 2020-03-03T03:00:28Z

Thanks @samhouts, I started testing that package in https://github.com/xamarin/xamarin-forms-samples/tree/master/UserInterface/Accessibility (Pulled as a zip from microsoft docs) and while the first page behaves as expected (with no tab indexes set). The tab-indexes page will allow the user to tab to the first button (Descending TabIndex), after which it gets stuck there.

For us, this is not an issue (as we don't intend to use TabIndex) but thought you'd prefer to know that.

joshuangfraedom · 2020-03-03T03:54:18Z

Xamarin.Forms.Platform.Android/VisualElementRenderer.cs

+			// use OS default--there's no need for us to keep going if there's one or fewer tab indexes!
+			if (tabIndexes.Count <= 1)
+				return base.FocusSearch(focused, direction);
+
 			int tabIndex = element.TabIndex;
 			AView control = null;
 			int attempt = 0;


@samhouts looking at the below code, it looks like the discovery of the next tab stop in the do while loop will only happen once, as maxAttempts = 0, and attempt starts at 0 but will be 1 at this stage, am I missing something?

Ah sorry missed the out in the extension

arevellfraedom · 2020-03-08T23:54:34Z

@samhouts hate to bring up the old "when will this be out", but.... when do you expect this to be out? We need to rely on this change for our upcoming release, which realistically means it being part of a 4.5 patch with 1-2 months. If this is not realistic, can you give an estimate of when it may be part of an official release so we may plan?

Assuming it will indeed be merged to 4.5 and not 4.4

samhouts · 2020-03-09T16:50:24Z

@arevellfraedom So, first, whenever we merge something to a release branch like 4.4.0, it gets merged up through all the active releases and then to master. Thus it will be in 4.4.0, 4.5.0, and 4.6.0, and in the nightlies from master.

Second, I can't make any promises about release dates, but yes, this has a high probability of being released within two weeks. 🤞

arevellfraedom · 2020-03-09T19:15:15Z

That would be amazing and I highly appreciate the effort. This change is very positive and will improve the accessibility of probably 90% of Xamarin forms apps, which I’m sure those users affected will very much appreciate.

I will put some effort in shortly to look beyond the cases we don’t specifically need, but others might, to see if I can provide any more feedback to ensure this PR is as good as possible.

arevellfraedom · 2020-03-10T02:36:04Z

Hi @samhouts - I have done additional validation of the keyboard tabbing on Android and have a couple of observations.

These are all using https://github.com/xamarin/xamarin-forms-samples/tree/master/UserInterface/Accessibility:

Bold = fail
Italic = pass

4.4.0.993041

Ascending tabindex

Tabbing from nav bar: gets stuck on "Ascending tabindex" button
If manually tap in "Enter data", then can tab through fields until "Select a Monkey" after which point it is impossible to proceed
If skip to "Enter search term" then can tab through the last three elements, but then which can't be tabbed through the same as "Select a Monkey"

Descending tabindex

Same pass/fail as ascending (can't exit nav bar etc), just in reverse

All tabindex = 0

Can tab from nav bar and through buttons
Can't tab through Date or "Select a Monkey"
On wrap around, skips buttons

Toggle IsTabStop

No fields can be tabbed through
Ignored if all tabindex = 0

Alternating IsTabStop

Tabbing from nav bar: gets stuck on "Ascending tabindex" button
Ignored if all tabindex = 0
If "Enter data" tapped, then "Enter name" correctly skipped on tab

4.4.0.991757

Tabbing from nav bar: gets stuck on some outer element and does not get into buttons
If manually tap in "Enter data", then can tab through fields until "Select a Monkey" after which point it is impossible to proceed
If skip to "Enter search term" then can tab through the last three elements, but then which can't be tabbed through the same as "Select a Monkey"

Descending tabindex

Same pass/fail as ascending (can't exit nav bar etc), just in reverse

All tabindex = 0

Tabbing from nav bar: gets stuck on some outer element and does not get into buttons
Can't tab through Date or "Select a Monkey"
On wrap around, skips buttons

Toggle IsTabStop

No fields can be tabbed through
Still works if tabindex = 0

Alternating IsTabStop

Tabbing from nav bar: gets stuck on some outer element and does not get into buttons
Still works if tabindex = 0
If "Enter data" tapped, then "Enter name" correctly skipped on tab

3.6.0

Ascending tabindex

Can tab from nav bar and through buttons
Tabbing through "Select a Monkey" tabs to next field after popup value selected
On wrap, buttons are tabbed through then fields

Descending tabindex

Same passes as Ascending

All tabindex = 0

Same passes as Ascending

Toggle IsTabStop

Fields cannot be tabbed through

Alternating IsTabStop

Same passes as Ascending

Summary
3.6.0 was basically perfect
4.4.0.991757 had particular problems getting from the navbar and when tabindex = 0
4.4.0.993041 corrects behaviour when tabindex = 0, but still struggles otherwise, in particular gets stuck on the first button, doesn't allow fields such as "Select a Monkey" to be tabbed through and also has a regression when tabindex = 0 but tabstop = false

4.4.0.993041 is still good for us, but does restore full 3.6.0 functionality

samhouts · 2020-03-13T16:18:26Z

@arevellfraedom Thank you so much for the detailed test and review!! Given this does improve behavior, we'll merge it, and then we'll continue working on the remaining issues you mention.

The basic problem is that once we override the accessibility tree in iOS, we're stuck with that, even if we don't need to alter it. I'm having trouble finding a way to accurately reproduce it, since iOS doesn't seem to expose it. If ANYONE has has success with that, I'd love to hear what you did!

Thanks to EVERYONE who has been pushing this area forward, either by your reports, testing, review, etc!!

arevellfraedom · 2020-03-13T19:21:19Z

@arevellfraedom Thank you so much for the detailed test and review!! Given this does improve behavior, we'll merge it, and then we'll continue working on the remaining issues you mention.

The basic problem is that once we override the accessibility tree in iOS, we're stuck with that, even if we don't need to alter it. I'm having trouble finding a way to accurately reproduce it, since iOS doesn't seem to expose it. If ANYONE has has success with that, I'd love to hear what you did!

Thanks to EVERYONE who has been pushing this area forward, either by your reports, testing, review, etc!!

@samhouts , if I understand the issue you are facing with iOS correctly, I may be able to help.
You are correct that once you implement the UIAccessibilityContainer protocol (through exporting accessibilityElementCount etc) you are stuck with overriding the children of that view for accessibility. Once you have implemented the protocol, you can't get back to asking the OS to do it instead, at least not without obj-c method swizzling.

I see two options:

Have the PageContainer (the one implementing the protocol) be the View of the VC conditional on whether you want to override the accessibility tree. Not sure how practical this is but it would be the most flexible option (I think)
Use the less known fact you can call the setAccessibilityElements method of UIAccessibilityContainer.
So the strange thing about this protocol is that you can either act by responding as a getter (whenever the OS feels it needs to know the accessibility children) or simply set them whenever you choose. In many way, acting as a getter is easier and more reliable (as it is called at the right times and can respond with elements based on the situation), but it has the major downside, as you have seen, that it is impossible to reply with "do the default". That is what setAccessibilityElements can be useful for.

So to make this useful, the first thing to determine is when to call it - the good news is that the answer is "any time before voiceover scans the page". So ViewDidLoad for example is just fine. So simply go through the logic of TabStop and TabIndex, and when you are ready, call setAccessibilityElements with the children or simply don't call it if you don't want to override. I think sending null might even remove an override, but haven't checked.

The second thing is to know how to call it - this is not so easy as this method is so strange it does not appear on the IUIAccessibilityContainer interface. What you need to do instead is:

Create a selector for it
SetAccessibilityElementsSelector = new Selector("setAccessibilityElements:");
Prepare a NSMutableArray<object> of the child elements you wish to set
Prepare a way of sending a message to the container you want to set the children on
[DllImport("/usr/lib/libobjc.dylib", EntryPoint = "objc_msgSend")]
private static extern void void_objc_msgSend_IntPtr(IntPtr receiver, IntPtr selector, IntPtr arg1);
Send the message
void_objc_msgSend_IntPtr(containingView.Handle, SetAccessibilityElementsSelector.Handle, array?.Handle ?? IntPtr.Zero);
This is probably most usefully exposed as an extension method with signature
public static void SetAccessibilityElements<TElement>([NotNull] this UIView containingView, [CanBeNull, ItemNotNull] IEnumerable<TElement> childElements) where TElement : NSObject

Hopefully I have understood the question and that gives you some new options.

bjarteskogoy · 2020-03-26T13:07:29Z

Will this show up in Xamarin.Forms 4.5 soon?

PureWeen · 2020-03-26T16:47:08Z

This should be available in the nuget that went out yesterday.

@samhouts correct me if I'm wrong

samhouts · 2020-03-26T18:22:38Z

Yep, that's correct. It's now available in the latest 4.4.0, 4.5.0, and 4.6.0 releases. Thanks!

rnic99 · 2020-03-31T19:29:01Z

@samhouts Im not seeing any major difference in accessibility behavior on the latest stable of 4.5, however I am seeing a significant improvement on the 4.6 pre-release

samhouts · 2020-03-31T20:45:09Z

@rnic99 You tried 4.5.0.495 and there was no difference?

rnic99 · 2020-03-31T20:47:09Z

@samhouts Yep pulled down 4.5.0.495 this morning and the behavior was identical. I should also point out I was testing on an iPhone 6 with iOS version 12 and a OnePlus 3t with android 9.

arevellfraedom · 2020-03-31T20:57:56Z

It would appear 4.5 SR2 was taken shortly before this PR was merged, although I cannot be sure

samhouts · 2020-03-31T21:01:49Z

4.5.0.495 was release-4.5.0-sr2.2, and this commit is in it. That's strange.

rnic99 · 2020-03-31T21:27:46Z

@samhouts I just re-downloaded the stable release and it seems to be working fine on Android, and iOS. Or atleast its showing the same behavior as it was in 4.6 pre. My apologies for the confusion. I have a feeling there was a problem where even though it was showing 4.5 under my XF packages it was still deploying as 4.4.

samhouts · 2020-03-31T21:38:57Z

@rnic99 Phew! I was hoping that was the case. Thanks for checking again!!

rnic99 · 2020-03-31T21:40:01Z

@samhouts of course, sorry for causing distress

samhouts and others added 11 commits February 20, 2020 15:35

add nifty test page

9e41236

TabIndexes will not skip layouts; will skip disabled/unfocusable views

4ffd971

[iOS|Android] Standardize logic, don't duplicate TabIndex logic

cf7e65c

fix test case

3019242

[iOS] Make Frame a tab stop

2cf229b

Update ShellFlyoutRenderer.cs

8100599

[iOS] Hide Master page from Accessibilty when not presented

aba8941

fixes #4220

final test case should work without so much explict tabindexing!

79f6258

[iOS] Single or no TabIndex = default page order

67d02a4

No tabindex? No problem!

c75f37e

[Android] Single or no TabIndex = default page order

e679c57

samhouts requested a review from StephaneDelcroix as a code owner February 25, 2020 01:05

samhouts linked an issue Feb 25, 2020 that may be closed by this pull request

[Bug] Accessibility Show Focus Traps or Out-Of-Order Reading #9653

Closed

samhouts added ControlGallery p/iOS 🍎 p/Android a/shell 🐚 a/a11y 🔍 i/high Completely doesn't work, crashes, or is unusably slow, has no obvious workaround; occurs less often t/bug 🐛 m/high impact ⬛ partner/cat 😻 labels Feb 25, 2020

samhouts linked an issue Feb 25, 2020 that may be closed by this pull request

iOS VoiceOver/Android TalkBack reads elements on Master menu page when scrolling over elements on Detail page #4220

Closed

This comment has been minimized.

Sign in to view

joshuangfraedom reviewed Mar 3, 2020

View reviewed changes

samhouts mentioned this pull request Mar 3, 2020

[Bug] Xamarin.Forms TabIndex not working on Android and iOS #7568

Closed

samhouts merged commit 77e411d into 4.4.0 Mar 13, 2020

samhouts deleted the fix-9653 branch March 13, 2020 16:38

samhouts added this to the 4.4.0 milestone Mar 20, 2020

PureWeen mentioned this pull request Jun 17, 2020

Convert to using list of accessibility elements for iOS tab indexes #11077

Merged

2 tasks

HelenMamalaki mentioned this pull request Sep 21, 2020

[Bug] Android VoiceOver does not read DetailPage inside a MasterDetailPage #12195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[iOS|Android] Fix screen reader order #9702

[iOS|Android] Fix screen reader order #9702

samhouts commented Feb 25, 2020 •

edited

Loading

This comment has been minimized.

arevellfraedom commented Mar 3, 2020

joshuangfraedom Mar 3, 2020

joshuangfraedom Mar 3, 2020

arevellfraedom commented Mar 8, 2020 •

edited

Loading

samhouts commented Mar 9, 2020

arevellfraedom commented Mar 9, 2020

arevellfraedom commented Mar 10, 2020 •

edited

Loading

samhouts commented Mar 13, 2020

arevellfraedom commented Mar 13, 2020 •

edited

Loading

bjarteskogoy commented Mar 26, 2020

PureWeen commented Mar 26, 2020

samhouts commented Mar 26, 2020

rnic99 commented Mar 31, 2020 •

edited

Loading

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020 •

edited

Loading

arevellfraedom commented Mar 31, 2020

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020

[iOS|Android] Fix screen reader order #9702

[iOS|Android] Fix screen reader order #9702

Conversation

samhouts commented Feb 25, 2020 • edited Loading

Description of Change

Issues Resolved

API Changes

Platforms Affected

Behavioral/Visual Changes

Before/After Screenshots

Testing Procedure

PR Checklist

This comment has been minimized.

arevellfraedom commented Mar 3, 2020

joshuangfraedom Mar 3, 2020

Choose a reason for hiding this comment

joshuangfraedom Mar 3, 2020

Choose a reason for hiding this comment

arevellfraedom commented Mar 8, 2020 • edited Loading

samhouts commented Mar 9, 2020

arevellfraedom commented Mar 9, 2020

arevellfraedom commented Mar 10, 2020 • edited Loading

samhouts commented Mar 13, 2020

arevellfraedom commented Mar 13, 2020 • edited Loading

bjarteskogoy commented Mar 26, 2020

PureWeen commented Mar 26, 2020

samhouts commented Mar 26, 2020

rnic99 commented Mar 31, 2020 • edited Loading

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020 • edited Loading

arevellfraedom commented Mar 31, 2020

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020

samhouts commented Mar 31, 2020

rnic99 commented Mar 31, 2020

samhouts commented Feb 25, 2020 •

edited

Loading

arevellfraedom commented Mar 8, 2020 •

edited

Loading

arevellfraedom commented Mar 10, 2020 •

edited

Loading

arevellfraedom commented Mar 13, 2020 •

edited

Loading

rnic99 commented Mar 31, 2020 •

edited

Loading

rnic99 commented Mar 31, 2020 •

edited

Loading