[Android] TalkBack doesn't work on the right screen on dual screen smartphone

[lookuper]

Steps to Reproduce

Reproducible on any flutter app.

1. Open flutter app on dual screen device (Surface Duo 1/2 for example)
2. Move the app to the right screen
3. Enable Android TalkBack (screen narrator)
4. Try to click/move focus to any item in the app.

When the app open on left the screen, or in dual screen mode, TalkBack works as expected.

flutter doctor -v

[✓] Flutter (Channel stable, 2.5.3, on macOS 12.0.1 21A559 darwin-x64, locale en-US)
    • Flutter version 2.5.3 
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (7 weeks ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/.../Library/Android/sdk
    • Platform android-31, build-tools 30.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 13.1, Build version 13A1030d
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.62.3)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.29.0

[✓] Connected device (3 available)
    • Maksym’s iPhone (mobile) • 00008101-000E61993684001E • ios            • iOS 15.0.2 19A404
    • macOS (desktop)          • macos                     • darwin-x64     • macOS 12.0.1 21A559 darwin-x64
    • Chrome (web)             • chrome                    • web-javascript • Google Chrome 96.0.4664.55

• No issues found!

[maheshmnj]

Hi @lookuper, Thanks for filing the issue. Please share your output of flutter doctor -v on which the app was built.

[lookuper]

Hi @lookuper, Thanks for filing the issue. Please share your output of flutter doctor -v on which the app was built.

Added. Also any Android app supports left/right/dual screen TalkBack function as expected.

[alcrus]

Is there any estimate when it will be fixed? Bug was opened almost year ago.

[andreidiaconu]

Writing a few of my findings here. I have found two separate bugs while investigating this:

Steps to reproduce:

• Make sure TalkBack is inactive.
• Start the app on the right screen (hello world from flutter create . should do)
• Start TalkBack using a hardware shortcut (to ensure no UI refresh is triggered by touch)
• Use a gesture shortcut to make TalkBack dump what it sees to logs

I have seen two different behaviours unfold:

1. Node positions are out of the bounds of the Activity, making them invisible to TalkBack
   • Some node bounds are reported on the left screen (offset is 0,0) and because the root node and the activity are reported on the right screen (offset is 1410, 0), TalkBack considers them invisible.
   • Note that sometimes some of the nodes are reported correctly. The first node considered a "View" is always reported wrong, and because that is the root of everything, they are all considered invisible even when the numbers seem right for an individual node.
   • TalkBack also complains that there are focusable nodes that are not visible (Focus, is focusable and has no visible children: prints node information)
   • I suspect that this bug can be reproduced on tablets or other devices with multi-window support.

2. The Accessibility Bridge has an empty Semantics tree.
   • The logs from TalkBack only have a root view (corresponding to FlutterView) with no children.
   • The Flutter semantics tree is communicated to the Flutter accessibility bridge only if accessibility is enabled. This is by design.
   • From debugging, I noticed that the semantics map cached inside the accessibility bridge is empty in some situations when it should be populated.
   • Seems like enabling accessibility does not also trigger a Semantics tree update in certain situations. I have yet to reliably reproduce the behavior.
   • This version of the bug should be reproducible on any Android device, although I did only test on a Surface Duo.

Regardless of the exact way it fails, using the steps to reproduce on a Surface Duo always ends up providing no accessibility for users.
For behavior no 1, at this point I am not sure if there is something unusual happening on the Surface Duo or if this can be reproduced using a tablet with multi-window support. Testing on a tablet should shed light on that.
For behavior no 2, it looks like a race condition in the way that accessibility is both enabled in the bridge and the semantics is populated and should be reproducible on any Android device.