Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for UIA with chromium based browsers #12025

merged 17 commits into from Feb 3, 2021


Copy link

@feerrenrut feerrenrut commented Feb 2, 2021

This is based off of PR "Add advanced setting to allow UIA in chromium based browsers #11079"

However, due to the large amount of moved code, NV Access has decided not to squash this PR. Instead the commits have been cleaned up to make following the history as easy as possible. The changes from #11079 are as follows:

  • Code is moved between files verbatim, in commits independent from any other changes. It should be noted that these commits will not run, but serve to make diffs / blame easier to follow.
  • Linting is done in independent commits. This allows logic changes to be differentiated from "clean up"
  • UIA/edge becomes UIA/spartan_edge and a placeholder is added for UIA/anaheim_edge

This PR supersedes and thus closes #11079, however the history there is still relevant and useful.

Link to issue number:

Related to #10555

Summary of the issue:

Chromium based browsers now contain a UIA implementation which may be exposed (E.G. in Edge Anaheim). Allowing this to be used will allow developers to work to stabilize and mature this technology. For users on systems where process injection is not possible, access to UIA in Edge can act as a fallback option.

Description of how this pull request fixes the issue:

Provides an advanced setting to use UIA when available. Moves significant portions of the UIA code for reuse:

  • Web
  • Legacy Edge (Spartan)
  • Chromium based browsers such as Edge Anaheim

Testing performed:

Manual testing.

Known issues with pull request:


Change log entry:

New features

- Early support for UIA with Chromium based browsers (such as Edge).

Copy link

Copy link

I'm slightly worried about introducing underscores in the names here, as it is not following the naming conventions we usually use.

Copy link

michaelDCurran commented Feb 2, 2021 via email

michaelDCurran previously approved these changes Feb 3, 2021
Copy link

@michaelDCurran michaelDCurran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have done the same manual testing I did on the original pr and all was good for modern Edge.
I hit an issue in classic edge, but this was actually introduced by pr #11969 so I have separately opened pr #12026 to address that. But other than that things seemed to be working well.

Copy link
Contributor Author

very clearly denote very different implementations of Edge. Especially when they are living in the same directory.

Hmm perhaps there should be more structure here eg "namespaces". Consider if we moved the files:

  |- New Package: Web
  |   |- New Package: Chromium
  |   |   |-
  |   |-

Copy link

michaelDCurran commented Feb 3, 2021 via email

@@ -417,6 +417,9 @@ def _get_name(self):
return ""

#: Type definition for auto prop '_get_role'
role: str
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this annotation is correct since roles are represented by ROLE_ constants in controlTypes and they are integers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right @lukaszgo1, good catch.

Copy link
Contributor Author

I've been looking at the layout of this code and reconsidering whether a nested structure is better. I think it is better for this code to stick with a flatter structure, otherwise there are dependencies going in the wrong direction:

  • chromium depends on Web
  • Web depends on UIA
  • spartan_edge depends on UIA
    Nesting these as packages requires requires multiple level relative imports or absolute imports.

I will however rename spartan_edge and anaheim _edge to spartanEdge and anaheimEdge respectively.

feerrenrut and others added 16 commits February 3, 2021 16:56
UIA.Web is a new base for browser specific implementations:
- spartan-edge
- Chromium based browsers: anaheim-edge, chrome

This commit is a direct move (no alterations) of code to preserve history.

- Move function splitUIAElementAttribs
- Move EdgeTextInfo
- Move shareable aspects of class EdgeNode:
  * _get_role
  * _get_states
  * _get_ariaProperties
  * _get_isCurrent
  * _get_roleText
  * _get_placeholder
  * _get_landmark
- Move class EdgeList
- Move class EdgeHeadingQuickNavItem
- Move class EdgeHeadingQuicknavIterator
- Move shareable aspects of class EdgeHTMLTreeInterceptor:
  * makeTextInfo
  * shouldPassThrough
  * _iterNodesByType
This is a module containing common classes above the browser specific
classes (, The browser specific classes inherit from
these web module classes.

- Move splitUIAElementAttribs(attribsString) from edge module to web module
- Move common implementation of edge.EdgeNode and rename to web.UIAWeb(UIA),
  edge.EdgeNode now inherits from web.UIAWeb.
- Move implementation of edge.EdgeList and rename to web.List(UIAWeb),
  edge.EdgeList now inherits from web.List
- Move implementation of edge.EdgeHTMLTreeInterceptor and rename to
  web.UIAWebTreeInterceptor, edge.EdgeHTMLTreeInterceptor now inherits from
The concrete implementation relied on knowing that self is
actually a subclass and providing varied behaviour.

Instead, abstract this test so it can be altered
via polymorphism.
Use chromium treeinterceptor

Use Web module for chromium module

Use chromium.ChromiumUIADocument in findOverlayClasses
Trade off: slight degradation of performance.

Test each child to see if it is clipped by or lives completely
outside the parent text range, and if so appropriately clip
the child range and break out of the for loop.

Previously these checks only happened on the final child to try
and avoid extra calls, but Edge sometimes returns many more
children then it should when on the end of a list, and a trivial
slow down is much preferred over a 20 second freeze.
Trade-off: only the first line is fetched.

If the given object should have its text presented (E.g. a document
or edit field) and the line at the selection cannot be fetched, then
only present the first line.

Previously the text for the entire object would be fetched, which in
the case of Edgium, was extremely costly and could cause a freeze of
10 seconds or more.
As headings are exposed in the element tree,
don't expose them in format fields.
skip children that appear completely before the start of the parent textRange.
Stops a freeze on some checkboxes in Edgium and is generally safer.
Make sure interactive / presentational lists are presented
correctly in browseMode in Edgium.
… with UIA

Embedded object characters in Edgium should not stop overriding labels from being used.
Allows ignoring of layout tables when configured to do so.
Ensure the value of comboboxes are used as content.
Names of controls (that do not use their name as content)
are always reported:
 Currently no way to tell if author has explicitly set name. Therefore
 always report the name if the control is not of a type that by
 definition uses its name for content. This may cause some duplicate
 speaking, but that is currently better than nothing at all.
PR #12025 was not squash merged (which is the usual practice for this project).
Instead, to preserve the history of moved then modified code, the commits have
been rebased directly onto master.
@feerrenrut feerrenrut merged commit 11ba984 into master Feb 3, 2021
@feerrenrut feerrenrut deleted the UiaWithChromium branch February 3, 2021 09:38
@nvaccessAuto nvaccessAuto added this to the 2021.1 milestone Feb 3, 2021
michaelDCurran pushed a commit that referenced this pull request Apr 21, 2021

Fixes #12114

PR #12025 started catching only very specific exceptions when getting selection of edit fields. However in Firefox attempting to get caret for non focused edit fields results in RuntimeError which made it impossible to speak these controls.

Description of how this pull request fixes the issue:
When getting content of edit fields in speech, we're now catching RuntimeError as well and treating this situation like no selection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

6 participants